2016-11-16T13:41:08.617-0500 I CONTROL [main] 2016-11-16T13:41:08.617-0500 I CONTROL [main] ** WARNING: Insecure configuration, access control is not enabled and no --bind_ip has been specified. 2016-11-16T13:41:08.617-0500 I CONTROL [main] ** Read and write access to data and configuration is unrestricted, 2016-11-16T13:41:08.617-0500 I CONTROL [main] ** and the server listens on all available network interfaces. 2016-11-16T13:41:08.617-0500 I CONTROL [main] 2016-11-16T13:41:08.618-0500 I SHARDING [mongosMain] MongoS version 3.2.5 starting: pid=8729 port=29101 64-bit host=chi-ppe-oas001 (--help for usage) 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] db version v3.2.5 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] git version: 34e65e5383f7ea1726332cb175b73077ec4a1b02 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] allocator: tcmalloc 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] modules: none 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] build environment: 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] distmod: rhel62 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] distarch: x86_64 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] target_arch: x86_64 2016-11-16T13:41:08.618-0500 I CONTROL [mongosMain] options: { net: { port: 29101 }, processManagement: { fork: true, pidFilePath: "/data_mount/mongodata/mongos/mongos.pid" }, sharding: { configDB: "csReplSet/bxb-ppe-oas002:29102,chi-ppe-oas019:29102,bxb-ppe-oas012:29102" }, systemLog: { destination: "file", path: "/data_mount/mongodata/mongos/mongos.log", quiet: true } } 2016-11-16T13:41:08.618-0500 I SHARDING [mongosMain] Updating config server connection string to: csReplSet/bxb-ppe-oas002:29102,chi-ppe-oas019:29102,bxb-ppe-oas012:29102 2016-11-16T13:41:08.618-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:41:08.618-0500 I NETWORK [ReplicaSetMonitorWatcher] starting 2016-11-16T13:41:08.620-0500 I SHARDING [thread1] creating distributed lock ping thread for process chi-ppe-oas001:29101:1479321668:-284942927 (sleeping for 30000ms) 2016-11-16T13:41:08.784-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:08.784-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:08.866-0500 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to bxb-ppe-oas002:29102 2016-11-16T13:41:09.368-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:09.368-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:09.951-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:09.951-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:10.536-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:10.536-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:10.863-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: InterruptedAtShutdown: could not get updated shard list from config server due to interrupted at shutdown 2016-11-16T13:41:11.092-0500 I NETWORK [replSetDistLockPinger] Detected bad connection created at 1479321668676267 microSec, clearing pool for bxb-ppe-oas002:29102 of 0 connections 2016-11-16T13:41:11.121-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:11.121-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:12.336-0500 I NETWORK [replSetDistLockPinger] Socket recv() errno:104 Connection reset by peer 10.170.7.6:29102 2016-11-16T13:41:12.336-0500 I NETWORK [replSetDistLockPinger] SocketException: remote: (NONE):0 error: 9001 socket exception [RECV_ERROR] server [10.170.7.6:29102] 2016-11-16T13:41:12.365-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:12.365-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:12.944-0500 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to bxb-ppe-oas012:29102 2016-11-16T13:41:12.949-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:12.976-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:12.976-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:13.534-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:13.590-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:13.590-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:14.174-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:14.201-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:14.201-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:14.788-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:14.817-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:14.817-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:15.374-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:15.430-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:15.430-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:15.988-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:16.043-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:16.043-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:16.599-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:16.655-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:16.655-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:17.169-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: InterruptedAtShutdown: could not get updated shard list from config server due to interrupted at shutdown 2016-11-16T13:41:17.184-0500 I NETWORK [replSetDistLockPinger] Detected bad connection created at 1479321668678219 microSec, clearing pool for bxb-ppe-oas012:29102 of 0 connections 2016-11-16T13:41:17.238-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:17.265-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:17.265-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:17.265-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:41:17.794-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:18.570-0500 I NETWORK [replSetDistLockPinger] Socket recv() errno:104 Connection reset by peer 10.170.7.16:29102 2016-11-16T13:41:18.571-0500 I NETWORK [replSetDistLockPinger] SocketException: remote: (NONE):0 error: 9001 socket exception [RECV_ERROR] server [10.170.7.16:29102] 2016-11-16T13:41:18.626-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:18.626-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:18.626-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:41:19.184-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:19.196-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:19.239-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:19.239-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:19.239-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:41:19.239-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csReplSet 2016-11-16T13:41:19.768-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:19.824-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:19.879-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:19.879-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:19.879-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:41:20.408-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:20.464-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:20.520-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:20.520-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:20.520-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:41:21.075-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:21.103-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:21.158-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:21.158-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:21.158-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:41:21.685-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:21.743-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:21.800-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:21.800-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:21.800-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:41:22.330-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:22.385-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:22.443-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:22.443-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:22.443-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:41:22.998-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:23.054-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:23.083-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:23.083-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:23.083-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:41:23.640-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:23.669-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:23.725-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:23.725-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:23.725-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:41:24.280-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:24.338-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:24.365-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:24.365-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:24.365-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:41:24.921-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:24.950-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:25.007-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:25.007-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:25.007-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:41:25.563-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:25.618-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:25.645-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:25.645-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:25.645-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:41:26.173-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:26.226-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:26.280-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:26.280-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:26.280-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:41:26.834-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:26.863-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:26.917-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:26.917-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:26.917-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:41:27.444-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:27.499-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:27.557-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:27.557-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:27.557-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:41:28.112-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:28.168-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:28.195-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:28.195-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:28.195-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:41:28.682-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:28.709-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:28.749-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:28.749-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:28.749-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:41:29.276-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:29.332-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:29.389-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:29.389-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:29.389-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:41:29.947-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:30.002-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:30.031-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:30.031-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:30.031-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:41:30.031-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:41:32.060-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:32.118-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:32.175-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:32.175-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:32.175-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:41:32.729-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:32.756-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:32.812-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:32.812-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:32.812-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:41:33.341-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:33.396-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:33.454-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:33.454-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:33.454-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:41:34.010-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:34.037-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:34.092-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:34.092-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:34.093-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:41:34.622-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:34.675-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:34.729-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:34.729-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:34.729-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:41:35.285-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:35.339-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:35.368-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:35.368-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:35.368-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:41:35.897-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:35.951-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:36.008-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:36.008-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:36.008-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:41:36.564-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:36.617-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:36.646-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:36.646-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:36.646-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:41:37.204-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:37.259-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:37.288-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:37.288-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:37.288-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:41:37.844-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:37.898-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:37.927-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:37.927-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:37.927-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:41:37.927-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:41:38.749-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:41:39.927-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:41:39.927-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:41:41.957-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:42.010-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:42.066-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:42.066-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:42.066-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:41:42.593-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:42.647-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:42.704-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:42.704-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:42.704-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:41:43.258-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:43.314-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:43.343-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:43.343-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:43.343-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:41:43.900-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:43.928-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:43.981-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:43.981-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:43.981-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:41:44.510-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:44.566-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:44.620-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:44.620-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:44.620-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:41:45.147-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:45.204-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:45.262-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:45.262-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:45.262-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:41:45.817-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:45.873-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:45.902-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:45.902-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:45.902-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:41:46.431-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:46.485-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:46.539-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:46.539-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:46.539-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:41:47.066-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:47.124-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:47.181-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:47.181-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:47.181-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:41:47.735-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:47.791-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:47.820-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:47.820-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:47.820-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:41:48.376-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:48.405-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:48.462-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:48.462-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:48.462-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:41:48.805-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:48.834-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:48.891-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:48.891-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:41:48.891-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:41:49.018-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.073-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.102-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.102-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:49.102-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:41:49.266-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.322-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.379-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.379-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:49.379-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:41:49.631-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.685-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.739-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.739-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:49.739-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:41:49.937-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:49.992-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:50.019-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:50.019-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:50.019-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:41:50.293-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:50.320-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:50.373-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:50.373-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:50.373-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:41:50.573-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:50.600-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:50.656-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:50.656-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:50.656-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:41:50.929-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:50.958-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.016-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.016-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:51.016-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:41:51.185-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.243-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.298-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.298-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:51.298-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:41:51.569-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.599-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.656-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.656-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:51.656-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:41:51.852-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.905-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.935-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:51.935-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:51.935-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:41:52.210-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:52.264-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:52.293-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:52.293-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:52.293-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:41:52.464-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:52.517-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:52.573-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:52.573-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:52.573-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:41:52.848-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:52.875-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:52.931-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:52.931-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:52.931-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:41:52.931-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:41:53.129-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:53.156-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:53.209-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:53.209-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:53.209-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:41:53.765-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:53.821-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:53.850-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:53.850-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:53.850-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:41:54.379-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:54.435-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:54.490-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:54.490-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:54.490-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:41:54.960-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:55.016-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:55.046-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:55.046-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:41:55.046-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:41:55.575-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:41:55.603-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:41:55.629-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:41:55.629-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:41:55.629-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:41:55.629-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:41:55.629-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:41:57.629-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:41:58.892-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:41:59.629-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:41:59.629-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:01.657-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:01.712-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:01.768-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:01.768-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:01.768-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:42:02.297-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:02.352-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:02.408-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:02.408-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:02.408-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:42:02.965-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:02.995-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:03.052-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:03.052-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:03.052-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:42:03.581-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:03.638-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:03.694-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:03.694-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:03.694-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:42:04.221-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:04.277-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:04.333-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:04.333-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:04.333-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:42:04.888-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:04.944-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:04.973-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:04.973-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:04.973-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:42:05.530-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:05.584-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:05.611-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:05.611-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:05.611-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:42:06.165-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:06.192-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:06.250-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:06.250-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:06.250-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:42:06.805-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:06.859-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:06.886-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:06.886-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:06.886-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:42:07.442-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:07.496-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:07.523-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:07.523-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:07.523-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:42:08.052-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:08.106-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:08.163-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:08.163-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:08.163-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:42:08.719-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:08.748-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:08.802-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:08.802-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:08.802-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:42:08.949-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:09.006-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:09.033-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:09.034-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:42:09.034-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:42:09.331-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:09.386-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:09.442-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:09.442-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:09.442-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:42:09.998-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:10.053-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:10.082-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:10.082-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:10.082-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:42:10.610-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:10.665-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:10.719-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:10.719-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:10.719-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:42:11.275-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:11.330-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:11.359-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:11.359-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:11.359-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:42:11.888-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:11.944-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:11.997-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:11.997-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:11.997-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:42:12.555-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:12.582-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:12.638-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:12.638-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:12.638-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:42:12.638-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:42:14.693-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:14.751-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:14.780-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:14.780-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:14.780-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:42:15.336-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:15.389-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:15.416-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:15.417-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:15.417-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:42:15.972-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:16.028-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:16.057-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:16.057-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:16.057-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:42:16.613-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:16.640-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:16.693-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:16.693-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:16.694-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:42:17.251-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:17.280-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:17.335-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:17.336-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:17.336-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:42:17.863-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:17.918-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:17.974-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:17.974-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:17.974-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:42:18.503-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:18.557-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:18.612-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:18.612-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:18.612-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:42:19.089-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:19.140-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:19.145-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:19.145-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:42:19.145-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:42:19.674-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:19.730-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:19.785-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:19.785-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:19.785-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:42:20.341-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:20.398-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:20.425-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:20.425-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:20.425-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:42:20.979-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:21.037-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:21.066-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:21.066-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:21.066-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:42:21.066-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:23.066-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:25.066-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:25.629-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:27.067-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:29.067-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:29.145-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:42:31.067-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:42:31.067-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:33.094-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:33.150-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:33.206-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:33.206-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:33.206-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:42:33.761-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:33.790-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:33.844-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:33.844-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:33.844-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:42:34.400-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:34.427-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:34.481-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:34.481-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:34.481-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:42:35.035-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:35.062-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:35.117-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:35.118-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:35.118-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:42:35.647-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:35.704-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:35.760-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:35.760-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:35.760-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:42:36.315-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:36.373-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:36.402-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:36.402-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:36.402-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:42:36.958-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:37.013-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:37.040-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:37.040-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:37.040-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:42:37.596-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:37.653-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:37.680-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:37.680-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:37.680-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:42:38.234-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:38.263-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:38.318-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:38.319-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:38.319-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:42:38.846-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:38.902-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:38.957-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:38.957-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:38.957-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:42:39.200-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:39.258-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:39.287-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:39.287-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:42:39.287-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:42:39.511-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:39.538-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:39.594-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:39.594-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:39.594-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:42:40.123-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:40.181-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:40.234-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:40.234-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:40.234-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:42:40.788-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:40.817-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:40.871-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:40.871-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:40.871-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:42:41.425-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:41.452-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:41.508-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:41.508-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:41.508-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:42:42.035-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:42.091-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:42.146-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:42.146-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:42.146-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:42:42.704-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:42.759-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:42.786-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:42.787-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:42.787-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:42:43.316-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:43.373-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:43.431-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:43.431-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:43.431-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:42:43.986-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:44.042-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:44.071-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:44.071-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:44.071-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:42:44.071-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:42:46.125-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:46.152-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:46.208-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:46.208-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:46.208-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:42:46.764-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:46.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:46.846-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:46.846-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:46.846-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:42:47.374-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:47.431-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:47.487-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:47.487-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:47.487-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:42:48.016-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:48.071-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:48.129-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:48.129-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:48.129-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:42:48.685-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:48.738-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:48.765-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:48.765-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:48.765-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:42:49.321-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:49.342-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:49.350-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:49.350-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:49.350-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:42:49.906-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:49.933-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:49.989-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:49.989-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:49.989-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:42:50.516-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:50.574-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:50.629-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:50.629-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:50.629-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:42:51.156-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:51.212-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:51.268-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:51.268-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:51.268-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:42:51.797-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:51.852-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:51.908-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:51.908-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:51.908-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:42:52.462-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:42:52.491-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:42:52.546-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:42:52.547-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:42:52.547-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:42:52.547-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:54.547-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:55.629-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:56.547-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:58.547-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:42:59.350-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:43:00.548-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:43:00.548-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:02.603-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:02.632-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:02.688-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:02.688-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:02.688-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:43:03.215-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:03.271-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:03.324-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:03.324-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:03.324-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:43:03.854-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:03.911-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:03.968-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:03.968-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:03.968-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:43:04.526-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:04.553-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:04.609-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:04.609-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:04.609-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:43:05.163-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:05.190-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:05.245-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:05.245-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:05.245-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:43:05.801-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:05.856-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:05.885-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:05.885-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:05.885-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:43:06.439-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:06.466-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:06.520-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:06.520-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:06.520-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:43:07.076-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:07.131-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:07.158-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:07.158-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:07.158-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:43:07.686-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:07.743-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:07.796-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:07.797-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:07.797-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:43:08.352-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:08.381-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:08.437-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:08.437-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:08.437-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:43:08.964-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:09.022-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:09.075-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:09.075-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:09.075-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:43:09.408-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:09.465-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:09.494-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:09.494-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:43:09.494-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:43:09.631-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:09.660-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:09.716-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:09.716-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:09.716-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:43:10.245-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:10.302-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:10.360-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:10.360-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:10.360-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:43:10.915-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:10.973-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:11.000-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:11.000-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:11.000-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:43:11.554-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:11.581-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:11.635-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:11.635-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:11.635-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:43:12.189-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:12.218-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:12.272-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:12.272-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:12.272-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:43:12.799-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:12.855-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:12.912-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:12.912-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:12.912-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:43:13.468-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:13.523-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:13.550-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:13.550-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:13.550-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:43:13.551-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:43:15.606-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:15.635-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:15.689-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:15.689-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:15.689-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:43:16.217-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:16.272-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:16.328-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:16.328-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:16.328-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:43:16.857-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:16.911-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:16.966-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:16.967-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:16.967-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:43:17.496-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:17.553-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:17.607-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:17.607-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:17.607-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:43:18.136-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:18.193-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:18.251-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:18.251-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:18.251-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:43:18.805-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:18.858-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:18.887-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:18.887-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:18.887-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:43:19.441-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:19.469-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:19.526-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:19.526-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:19.526-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:43:20.082-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:20.137-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:20.165-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:20.165-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:20.165-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:43:20.719-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:20.748-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:20.805-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:20.805-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:20.805-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:43:21.359-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:21.417-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:21.446-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:21.446-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:21.446-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:43:22.001-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:22.030-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:22.086-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:22.086-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:22.086-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:43:22.086-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:24.086-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:25.629-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:26.087-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:28.087-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:29.526-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:43:30.087-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:43:30.087-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:32.115-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:32.170-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:32.226-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:32.226-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:32.226-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:43:32.780-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:32.835-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:32.864-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:32.864-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:32.864-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:43:33.420-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:33.449-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:33.503-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:33.503-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:33.503-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:43:34.060-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:34.118-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:34.145-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:34.145-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:34.145-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:43:34.674-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:34.728-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:34.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:34.783-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:34.783-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:43:35.341-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:35.396-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:35.425-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:35.425-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:35.425-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:43:35.981-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:36.008-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:36.066-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:36.066-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:36.066-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:43:36.620-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:36.649-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:36.704-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:36.704-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:36.704-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:43:37.260-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:37.316-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:37.343-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:37.343-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:37.343-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:43:37.870-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:37.926-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:37.981-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:37.981-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:37.982-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:43:38.539-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:38.595-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:38.624-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:38.624-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:38.624-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:43:39.151-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:39.207-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:39.262-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:39.262-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:39.262-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:43:39.580-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:39.609-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:39.665-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:39.665-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:43:39.665-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:43:39.818-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:39.845-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:39.902-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:39.903-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:39.903-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:43:40.460-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:40.516-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:40.543-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:40.543-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:40.543-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:43:41.070-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:41.126-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:41.181-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:41.181-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:41.181-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:43:41.737-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:41.764-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:41.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:41.820-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:41.820-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:43:42.373-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:42.401-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:42.458-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:42.458-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:42.458-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:43:43.014-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:43.043-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:43.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:43.100-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:43.100-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:43:43.100-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:43:45.158-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:45.187-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:45.242-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:45.242-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:45.242-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:43:45.770-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:45.825-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:45.883-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:45.883-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:45.883-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:43:46.438-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:46.494-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:46.521-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:46.521-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:46.521-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:43:47.050-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:47.106-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:47.163-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:47.163-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:47.163-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:43:47.719-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:47.748-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:47.804-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:47.804-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:47.804-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:43:48.361-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:48.390-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:48.447-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:48.448-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:48.448-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:43:49.001-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:49.055-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:49.084-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:49.084-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:49.084-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:43:49.638-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:49.694-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:49.694-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:49.694-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:49.694-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:43:50.223-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:50.279-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:50.334-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:50.334-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:50.334-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:43:50.890-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:50.945-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:50.973-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:50.973-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:50.973-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:43:51.500-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:43:51.556-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:43:51.609-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:43:51.609-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:43:51.609-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:43:51.609-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:53.610-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:55.610-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:55.630-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:57.610-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:59.610-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:43:59.694-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:44:01.611-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:44:01.611-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:03.638-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:03.694-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:03.748-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:03.748-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:03.748-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:44:04.275-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:04.329-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:04.387-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:04.387-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:04.387-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:44:04.914-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:04.969-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:05.025-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:05.025-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:05.025-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:44:05.554-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:05.611-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:05.669-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:05.669-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:05.669-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:44:06.222-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:06.278-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:06.307-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:06.307-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:06.307-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:44:06.863-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:06.920-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:06.949-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:06.949-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:06.949-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:44:07.478-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:07.536-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:07.589-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:07.589-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:07.589-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:44:08.145-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:08.199-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:08.228-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:08.228-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:08.228-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:44:08.755-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:08.811-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:08.866-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:08.866-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:08.866-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:44:09.424-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:09.451-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:09.506-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:09.507-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:09.507-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:44:09.750-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:09.807-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:09.836-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:09.836-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:44:09.836-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:44:10.036-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:10.089-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:10.143-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:10.143-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:10.143-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:44:10.699-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:10.728-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:10.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:10.783-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:10.783-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:44:11.341-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:11.396-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:11.425-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:11.425-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:11.425-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:44:11.982-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:12.038-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:12.065-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:12.065-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:12.065-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:44:12.592-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:12.648-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:12.703-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:12.703-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:12.703-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:44:13.231-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:13.288-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:13.344-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:13.344-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:13.344-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:44:13.898-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:13.955-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:13.982-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:13.982-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:13.982-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:44:14.509-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:14.565-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:14.619-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:14.619-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:14.619-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:44:14.619-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:44:16.675-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:16.704-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:16.757-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:16.757-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:16.757-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:44:17.315-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:17.372-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:17.401-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:17.401-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:17.401-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:44:17.957-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:18.011-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:18.040-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:18.040-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:18.040-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:44:18.569-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:18.624-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:18.680-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:18.680-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:18.680-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:44:19.235-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:19.292-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:19.322-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:19.322-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:19.322-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:44:19.875-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:19.890-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:19.903-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:19.903-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:19.903-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:44:20.432-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:20.485-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:20.543-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:20.543-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:20.543-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:44:21.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:21.157-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:21.184-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:21.184-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:21.184-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:44:21.738-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:21.767-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:21.823-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:21.823-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:21.823-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:44:22.350-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:22.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:22.459-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:22.460-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:22.460-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:44:22.987-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:23.042-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:23.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:23.100-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:23.100-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:44:23.100-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:25.100-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:25.630-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:27.100-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:29.101-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:29.903-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:44:31.101-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:44:31.101-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:33.130-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:33.188-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:33.243-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:33.243-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:33.243-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:44:33.797-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:33.851-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:33.878-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:33.878-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:33.878-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:44:34.407-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:34.462-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:34.520-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:34.520-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:34.520-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:44:35.076-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:35.130-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:35.159-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:35.159-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:35.159-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:44:35.713-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:35.740-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:35.795-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:35.795-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:35.795-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:44:36.351-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:36.404-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:36.433-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:36.433-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:36.433-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:44:36.962-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:37.020-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:37.077-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:37.077-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:37.077-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:44:37.606-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:37.662-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:37.717-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:37.717-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:37.717-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:44:38.271-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:38.298-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:38.354-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:38.354-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:38.354-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:44:38.883-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:38.938-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:38.995-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:38.996-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:38.996-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:44:39.553-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:39.582-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:39.636-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:39.636-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:39.636-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:44:39.956-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:40.010-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:40.039-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:40.039-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:44:40.039-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:44:40.163-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:40.217-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:40.272-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:40.272-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:40.272-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:44:40.826-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:40.883-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:40.910-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:40.910-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:40.911-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:44:41.468-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:41.523-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:41.552-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:41.552-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:41.552-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:44:42.108-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:42.135-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:42.189-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:42.189-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:42.189-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:44:42.745-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:42.800-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:42.829-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:42.829-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:42.829-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:44:43.357-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:43.414-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:43.471-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:43.471-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:43.471-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:44:44.025-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:44.054-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:44.112-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:44.112-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:44.112-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:44:44.112-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:44:46.169-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:46.225-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:46.254-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:46.254-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:46.254-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:44:46.809-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:46.837-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:46.892-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:46.892-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:46.892-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:44:47.420-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:47.473-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:47.527-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:47.527-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:47.527-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:44:48.083-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:48.140-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:48.169-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:48.169-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:48.169-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:44:48.696-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:48.752-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:48.806-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:48.806-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:48.806-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:44:49.363-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:49.421-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:49.448-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:49.448-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:49.448-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:44:50.004-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:50.033-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:50.088-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:50.088-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:50.088-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:44:50.642-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:50.698-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:50.727-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:50.727-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:50.727-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:44:51.282-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:51.311-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:51.369-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:51.369-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:51.369-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:44:51.924-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:51.980-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:52.007-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:52.007-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:52.007-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:44:52.563-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:44:52.592-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:44:52.647-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:44:52.647-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:44:52.647-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:44:52.647-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:54.647-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:55.630-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:56.648-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:44:58.648-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:00.088-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:45:00.648-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:45:00.648-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:02.702-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:02.760-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:02.787-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:02.787-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:02.787-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:45:03.317-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:03.380-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:03.435-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:03.435-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:03.435-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:45:03.993-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:04.022-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:04.077-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:04.077-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:04.077-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:45:04.631-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:04.687-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:04.717-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:04.717-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:04.717-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:45:05.275-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:05.329-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:05.356-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:05.356-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:05.356-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:45:05.913-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:05.941-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:05.998-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:05.998-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:05.998-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:45:06.527-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:06.581-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:06.635-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:06.635-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:06.635-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:45:07.188-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:07.244-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:07.271-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:07.271-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:07.271-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:45:07.828-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:07.857-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:07.914-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:07.914-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:07.914-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:45:08.470-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:08.525-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:08.554-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:08.554-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:08.554-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:45:09.112-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:09.141-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:09.197-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:09.197-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:09.197-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:45:09.754-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:09.810-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:09.837-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:09.837-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:09.837-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:45:10.142-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:10.198-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:10.227-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:10.227-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:45:10.227-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:45:10.392-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:10.450-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:10.477-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:10.477-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:10.477-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:45:11.033-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:11.060-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:11.114-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:11.114-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:11.114-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:45:11.642-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:11.697-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:11.754-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:11.754-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:11.755-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:45:12.284-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:12.337-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:12.393-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:12.393-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:12.393-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:45:12.920-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:12.976-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:13.031-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:13.031-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:13.031-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:45:13.589-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:13.616-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:13.670-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:13.670-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:13.670-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:45:13.670-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:45:15.727-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:15.781-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:15.810-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:15.810-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:15.810-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:45:16.364-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:16.393-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:16.447-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:16.447-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:16.447-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:45:16.974-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:17.030-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:17.087-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:17.087-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:17.087-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:45:17.614-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:17.672-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:17.727-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:17.727-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:17.727-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:45:18.282-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:18.311-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:18.367-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:18.367-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:18.367-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:45:18.922-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:18.952-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:19.009-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:19.009-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:19.009-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:45:19.537-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:19.594-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:19.652-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:19.652-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:19.652-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:45:20.181-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:20.235-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:20.281-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:20.281-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:45:20.281-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:45:20.834-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:20.888-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:20.917-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:20.917-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:20.917-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:45:21.444-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:21.498-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:21.554-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:21.554-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:21.554-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:45:22.111-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:22.167-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:22.196-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:22.196-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:22.196-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:45:22.196-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:24.196-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:25.630-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:26.196-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:28.197-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:30.197-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:30.281-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:45:32.197-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:45:32.197-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:34.253-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:34.282-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:34.337-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:34.337-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:34.338-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:45:34.893-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:34.947-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:34.976-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:34.976-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:34.976-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:45:35.505-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:35.563-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:35.618-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:35.618-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:35.618-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:45:36.176-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:36.205-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:36.262-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:36.262-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:36.262-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:45:36.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:36.849-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:36.906-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:36.906-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:36.906-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:45:37.464-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:37.491-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:37.545-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:37.545-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:37.545-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:45:38.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:38.128-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:38.181-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:38.181-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:38.181-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:45:38.735-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:38.793-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:38.820-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:38.820-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:38.820-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:45:39.374-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:39.403-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:39.459-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:39.459-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:39.459-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:45:40.014-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:40.068-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:40.097-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:40.097-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:40.098-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:45:40.310-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:40.364-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:40.419-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:40.419-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:45:40.419-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:45:40.651-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:40.680-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:40.736-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:40.736-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:40.736-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:45:41.291-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:41.347-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:41.376-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:41.376-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:41.376-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:45:41.903-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:41.957-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:42.012-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:42.012-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:42.012-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:45:42.540-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:42.595-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:42.651-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:42.651-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:42.651-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:45:43.205-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:43.234-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:43.291-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:43.291-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:43.291-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:45:43.847-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:43.904-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:43.933-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:43.933-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:43.933-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:45:44.462-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:44.516-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:44.572-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:44.572-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:44.572-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:45:45.127-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:45.157-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:45.214-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:45.214-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:45.214-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:45:45.214-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:45:47.270-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:47.328-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:47.355-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:47.355-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:47.355-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:45:47.911-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:47.968-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:47.995-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:47.995-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:47.996-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:45:48.553-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:48.609-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:48.638-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:48.638-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:48.638-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:45:49.167-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:49.221-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:49.276-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:49.276-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:49.276-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:45:49.834-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:49.891-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:49.920-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:49.920-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:49.921-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:45:50.448-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:50.475-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:50.503-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:50.503-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:50.503-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:45:51.033-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:51.089-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:51.143-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:51.143-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:51.143-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:45:51.699-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:51.728-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:51.785-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:51.785-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:51.785-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:45:52.343-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:52.398-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:52.426-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:52.426-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:52.426-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:45:52.953-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:53.007-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:53.063-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:53.063-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:53.063-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:45:53.592-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:45:53.647-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:45:53.701-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:45:53.701-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:45:53.701-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:45:53.701-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:55.630-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:55.701-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:57.702-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:45:59.702-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:00.504-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:46:01.702-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:46:01.702-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:03.756-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:03.812-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:03.841-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:03.841-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:03.841-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:46:04.368-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:04.424-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:04.482-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:04.482-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:04.482-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:46:05.037-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:05.065-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:05.120-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:05.120-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:05.120-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:46:05.676-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:05.731-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:05.760-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:05.761-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:05.761-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:46:06.288-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:06.343-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:06.399-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:06.399-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:06.399-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:46:06.957-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:06.984-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:07.039-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:07.039-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:07.039-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:46:07.567-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:07.622-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:07.678-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:07.678-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:07.678-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:46:08.205-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:08.261-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:08.317-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:08.317-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:08.317-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:46:08.872-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:08.899-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:08.953-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:08.953-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:08.953-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:46:09.511-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:09.538-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:09.595-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:09.595-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:09.595-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:46:10.151-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:10.180-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:10.236-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:10.236-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:10.236-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:46:10.559-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:10.586-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:10.644-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:10.644-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:46:10.644-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:46:10.765-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:10.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:10.876-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:10.876-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:10.876-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:46:11.432-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:11.486-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:11.513-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:11.513-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:11.513-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:46:12.042-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:12.097-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:12.153-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:12.153-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:12.153-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:46:12.709-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:12.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:12.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:12.793-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:12.793-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:46:13.351-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:13.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:13.435-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:13.435-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:13.435-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:46:13.991-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:14.018-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:14.074-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:14.074-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:14.074-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:46:14.631-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:14.687-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:14.716-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:14.716-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:14.716-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:46:14.716-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:46:16.772-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:16.801-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:16.854-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:16.854-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:16.854-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:46:17.384-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:17.441-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:17.495-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:17.495-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:17.495-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:46:18.050-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:18.078-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:18.131-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:18.132-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:18.132-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:46:18.661-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:18.716-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:18.774-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:18.774-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:18.774-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:46:19.327-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:19.355-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:19.408-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:19.408-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:19.409-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:46:19.936-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:19.993-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:20.049-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:20.049-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:20.049-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:46:20.603-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:20.661-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:20.671-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:20.671-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:46:20.671-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:46:21.201-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:21.254-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:21.310-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:21.310-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:21.310-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:46:21.837-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:21.893-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:21.950-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:21.950-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:21.950-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:46:22.478-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:22.533-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:22.589-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:22.589-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:22.589-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:46:23.117-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:23.175-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:23.232-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:23.232-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:23.232-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:46:23.232-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:25.233-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:25.630-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:27.233-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:29.233-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:30.672-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:46:31.233-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:46:31.233-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:33.289-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:33.343-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:33.372-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:33.372-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:33.372-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:46:33.927-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:33.983-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:34.010-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:34.010-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:34.010-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:46:34.566-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:34.620-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:34.647-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:34.647-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:34.647-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:46:35.174-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:35.230-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:35.285-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:35.285-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:35.285-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:46:35.814-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:35.868-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:35.924-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:35.924-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:35.924-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:46:36.453-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:36.510-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:36.566-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:36.566-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:36.566-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:46:37.123-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:37.152-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:37.208-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:37.208-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:37.208-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:46:37.764-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:37.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:37.848-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:37.848-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:37.848-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:46:38.377-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:38.433-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:38.489-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:38.489-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:38.489-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:46:39.045-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:39.098-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:39.126-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:39.126-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:39.126-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:46:39.681-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:39.710-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:39.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:39.766-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:39.766-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:46:40.320-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:40.349-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:40.404-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:40.405-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:40.405-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:46:40.699-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:40.755-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:40.808-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:40.808-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:46:40.809-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:46:40.960-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:41.016-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:41.043-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:41.043-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:41.043-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:46:41.597-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:41.654-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:41.684-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:41.684-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:41.684-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:46:42.213-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:42.270-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:42.326-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:42.326-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:42.326-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:46:42.853-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:42.911-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:42.965-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:42.965-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:42.965-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:46:43.520-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:43.548-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:43.601-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:43.601-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:43.601-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:46:44.157-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:44.184-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:44.238-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:44.238-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:44.238-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:46:44.238-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:46:46.292-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:46.347-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:46.376-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:46.376-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:46.376-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:46:46.930-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:46.987-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:47.015-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:47.015-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:47.015-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:46:47.544-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:47.598-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:47.652-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:47.652-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:47.652-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:46:48.207-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:48.235-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:48.292-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:48.292-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:48.292-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:46:48.822-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:48.875-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:48.929-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:48.929-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:48.929-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:46:49.485-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:49.540-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:49.568-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:49.568-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:49.568-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:46:50.097-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:50.152-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:50.210-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:50.210-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:50.210-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:46:50.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:50.795-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:50.852-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:50.852-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:50.852-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:46:51.410-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:51.465-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:51.495-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:51.495-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:51.495-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:46:52.052-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:52.106-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:52.135-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:52.135-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:52.135-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:46:52.691-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:46:52.718-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:46:52.774-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:46:52.774-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:46:52.774-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:46:52.774-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:54.774-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:55.631-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:56.774-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:46:58.774-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:00.775-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:00.852-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:47:02.775-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:47:02.775-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:04.831-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:04.860-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:04.917-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:04.917-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:04.917-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:47:05.473-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:05.500-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:05.554-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:05.554-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:05.554-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:47:06.083-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:06.141-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:06.195-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:06.195-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:06.195-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:47:06.724-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:06.782-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:06.837-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:06.837-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:06.837-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:47:07.393-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:07.450-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:07.480-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:07.480-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:07.480-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:47:08.033-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:08.089-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:08.118-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:08.118-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:08.118-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:47:08.674-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:08.701-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:08.759-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:08.759-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:08.759-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:47:09.313-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:09.342-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:09.396-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:09.396-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:09.396-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:47:09.923-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:09.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:10.031-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:10.031-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:10.031-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:47:10.586-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:10.615-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:10.669-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:10.669-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:10.669-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:47:10.908-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:10.935-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:10.989-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:10.989-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:47:10.989-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:47:11.225-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:11.252-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:11.306-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:11.306-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:11.306-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:47:11.835-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:11.893-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:11.950-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:11.950-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:11.950-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:47:12.506-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:12.535-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:12.591-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:12.591-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:12.591-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:47:13.146-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:13.173-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:13.229-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:13.229-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:13.229-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:47:13.785-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:13.838-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:13.866-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:13.866-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:13.866-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:47:14.422-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:14.449-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:14.504-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:14.504-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:14.504-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:47:15.034-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:15.089-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:15.147-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:15.147-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:15.147-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:47:15.676-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:15.732-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:15.788-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:15.788-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:15.788-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:47:15.788-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:47:17.844-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:17.873-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:17.927-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:17.927-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:17.927-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:47:18.483-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:18.510-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:18.564-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:18.564-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:18.564-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:47:19.093-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:19.151-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:19.209-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:19.209-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:19.209-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:47:19.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:19.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:19.847-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:19.847-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:19.847-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:47:20.374-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:20.428-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:20.484-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:20.484-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:20.484-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:47:21.038-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:21.044-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:21.067-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:21.067-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:21.067-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:47:21.624-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:21.680-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:21.707-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:21.707-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:21.707-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:47:22.234-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:22.290-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:22.344-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:22.344-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:22.344-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:47:22.900-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:22.927-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:22.982-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:22.982-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:22.982-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:47:23.538-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:23.567-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:23.624-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:23.624-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:23.624-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:47:24.180-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:24.236-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:24.265-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:24.265-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:24.265-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:47:24.265-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:25.631-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:26.265-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:28.265-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:30.266-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:31.067-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:47:32.266-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:47:32.266-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:34.322-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:34.377-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:34.404-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:34.404-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:34.404-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:47:34.959-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:35.015-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:35.042-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:35.042-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:35.042-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:47:35.598-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:35.625-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:35.681-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:35.681-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:35.681-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:47:36.237-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:36.264-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:36.322-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:36.322-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:36.322-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:47:36.877-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:36.933-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:36.960-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:36.960-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:36.960-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:47:37.514-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:37.570-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:37.597-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:37.597-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:37.597-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:47:38.155-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:38.209-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:38.236-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:38.236-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:38.236-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:47:38.790-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:38.846-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:38.873-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:38.873-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:38.873-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:47:39.401-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:39.456-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:39.510-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:39.510-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:39.510-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:47:40.038-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:40.093-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:40.149-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:40.149-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:40.149-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:47:40.705-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:40.760-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:40.789-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:40.789-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:40.789-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:47:41.123-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:41.152-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:41.209-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:41.209-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:47:41.209-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:47:41.343-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:41.372-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:41.428-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:41.428-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:41.428-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:47:41.983-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:42.041-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:42.068-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:42.068-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:42.068-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:47:42.624-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:42.678-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:42.707-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:42.707-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:42.707-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:47:43.263-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:43.319-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:43.348-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:43.348-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:43.348-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:47:43.876-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:43.932-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:43.988-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:43.988-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:43.988-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:47:44.542-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:44.569-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:44.625-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:44.625-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:44.625-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:47:45.182-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:45.236-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:45.265-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:45.265-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:45.265-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:47:45.265-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:47:47.321-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:47.349-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:47.404-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:47.404-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:47.404-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:47:47.960-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:47.987-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:48.041-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:48.041-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:48.041-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:47:48.568-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:48.624-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:48.678-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:48.678-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:48.678-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:47:49.232-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:49.259-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:49.315-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:49.315-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:49.315-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:47:49.844-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:49.901-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:49.955-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:49.955-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:49.955-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:47:50.511-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:50.569-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:50.596-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:50.596-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:50.596-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:47:51.125-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:51.181-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:51.236-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:51.236-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:51.236-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:47:51.766-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:51.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:51.875-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:51.875-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:51.875-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:47:52.431-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:52.460-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:52.514-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:52.514-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:52.514-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:47:53.041-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:53.099-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:53.156-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:53.156-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:53.156-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:47:53.712-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:47:53.741-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:47:53.795-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:47:53.795-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:47:53.795-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:47:53.795-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:55.631-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:55.795-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:57.795-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:47:59.795-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:01.237-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:48:01.796-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:48:01.796-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:03.825-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:03.879-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:03.936-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:03.936-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:03.936-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:48:04.492-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:04.546-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:04.573-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:04.573-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:04.573-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:48:05.128-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:05.156-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:05.213-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:05.213-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:05.213-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:48:05.740-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:05.796-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:05.853-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:05.853-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:05.853-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:48:06.407-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:06.436-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:06.494-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:06.494-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:06.494-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:48:07.051-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:07.080-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:07.136-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:07.136-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:07.136-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:48:07.665-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:07.721-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:07.778-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:07.778-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:07.778-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:48:08.333-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:08.362-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:08.415-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:08.416-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:08.416-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:48:08.943-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:08.998-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:09.052-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:09.052-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:09.052-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:48:09.606-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:09.636-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:09.693-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:09.693-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:09.693-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:48:10.249-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:10.276-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:10.332-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:10.332-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:10.332-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:48:10.859-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:10.913-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:10.970-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:10.970-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:10.970-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:48:11.266-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:11.323-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:11.379-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:11.379-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:48:11.379-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:48:11.498-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:11.555-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:11.611-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:11.611-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:11.611-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:48:12.166-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:12.220-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:12.247-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:12.247-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:12.247-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:48:12.801-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:12.829-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:12.884-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:12.884-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:12.884-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:48:13.442-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:13.497-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:13.526-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:13.526-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:13.526-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:48:14.054-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:14.111-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:14.165-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:14.165-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:14.165-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:48:14.694-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:14.749-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:14.803-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:14.803-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:14.803-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:48:14.803-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:48:16.861-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:16.890-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:16.947-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:16.947-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:16.947-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:48:17.505-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:17.534-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:17.591-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:17.591-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:17.591-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:48:18.147-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:18.202-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:18.231-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:18.231-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:18.231-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:48:18.759-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:18.814-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:18.870-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:18.870-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:18.870-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:48:19.425-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:19.481-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:19.510-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:19.510-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:19.510-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:48:20.066-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:20.120-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:20.147-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:20.147-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:20.147-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:48:20.677-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:20.732-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:20.788-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:20.788-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:20.788-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:48:21.345-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:21.374-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:21.430-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:21.430-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:21.430-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:48:21.985-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:22.039-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:22.068-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:22.068-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:22.068-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:48:22.624-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:22.653-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:22.711-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:22.711-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:22.711-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:48:23.268-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:23.324-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:23.351-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:23.351-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:23.351-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:48:23.351-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:25.352-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:25.631-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:27.352-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:29.352-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:31.352-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:31.430-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:48:33.353-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:48:33.353-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:35.410-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:35.466-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:35.493-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:35.493-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:35.493-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:48:36.048-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:36.104-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:36.133-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:36.133-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:36.133-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:48:36.687-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:36.741-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:36.770-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:36.770-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:36.770-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:48:37.297-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:37.351-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:37.405-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:37.405-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:37.405-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:48:37.960-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:38.016-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:38.045-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:38.045-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:38.045-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:48:38.599-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:38.656-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:38.685-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:38.686-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:38.686-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:48:39.241-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:39.299-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:39.326-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:39.326-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:39.326-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:48:39.882-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:39.911-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:39.964-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:39.964-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:39.965-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:48:40.520-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:40.549-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:40.605-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:40.605-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:40.605-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:48:41.159-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:41.217-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:41.246-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:41.246-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:41.246-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:48:41.459-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:41.515-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:41.568-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:41.568-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:48:41.568-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:48:41.802-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:41.859-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:41.886-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:41.886-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:41.886-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:48:42.442-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:42.469-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:42.525-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:42.525-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:42.525-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:48:43.082-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:43.140-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:43.169-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:43.169-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:43.169-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:48:43.698-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:43.755-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:43.811-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:43.811-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:43.811-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:48:44.367-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:44.424-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:44.453-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:44.453-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:44.453-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:48:44.982-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:45.040-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:45.097-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:45.097-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:45.097-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:48:45.655-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:45.710-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:45.739-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:45.739-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:45.739-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:48:46.295-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:46.322-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:46.378-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:46.378-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:46.378-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:48:46.378-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:48:48.434-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:48.490-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:48.517-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:48.517-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:48.517-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:48:49.046-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:49.104-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:49.159-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:49.159-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:49.159-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:48:49.689-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:49.742-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:49.798-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:49.798-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:49.798-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:48:50.327-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:50.383-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:50.436-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:50.437-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:50.437-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:48:50.992-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:51.048-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:51.075-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:51.075-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:51.075-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:48:51.596-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:51.629-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:51.651-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:51.651-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:48:51.651-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:48:52.207-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:52.261-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:52.288-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:52.288-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:52.288-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:48:52.844-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:52.898-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:52.925-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:52.925-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:52.925-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:48:53.481-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:53.535-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:53.564-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:53.564-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:53.564-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:48:54.119-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:54.147-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:54.202-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:54.202-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:54.202-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:48:54.732-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:48:54.789-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:48:54.845-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:48:54.845-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:48:54.845-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:48:54.845-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:55.631-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:56.845-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:48:58.845-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:00.846-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:01.652-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:49:02.846-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:49:02.846-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:04.875-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:04.933-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:04.989-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:04.989-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:04.989-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:49:05.518-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:05.571-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:05.627-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:05.627-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:05.627-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:49:06.183-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:06.210-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:06.266-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:06.266-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:06.266-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:49:06.824-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:06.851-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:06.908-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:06.908-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:06.908-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:49:07.464-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:07.520-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:07.547-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:07.547-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:07.547-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:49:08.076-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:08.132-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:08.187-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:08.187-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:08.187-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:49:08.745-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:08.800-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:08.829-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:08.829-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:08.829-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:49:09.385-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:09.441-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:09.470-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:09.470-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:09.470-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:49:09.997-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:10.055-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:10.111-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:10.111-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:10.111-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:49:10.666-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:10.722-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:10.751-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:10.751-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:10.751-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:49:11.307-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:11.334-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:11.390-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:11.390-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:11.390-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:49:11.707-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:11.736-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:11.794-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:11.794-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:49:11.794-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:49:11.919-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:11.975-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:12.030-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:12.030-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:12.030-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:49:12.588-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:12.617-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:12.671-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:12.671-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:12.671-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:49:13.225-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:13.254-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:13.309-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:13.309-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:13.310-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:49:13.837-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:13.892-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:13.950-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:13.950-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:13.950-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:49:14.508-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:14.537-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:14.590-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:14.590-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:14.590-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:49:15.146-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:15.200-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:15.227-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:15.227-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:15.227-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:49:15.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:15.838-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:15.866-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:15.866-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:15.866-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:49:15.866-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:49:17.922-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:17.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:18.006-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:18.006-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:18.006-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:49:18.562-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:18.616-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:18.643-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:18.643-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:18.643-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:49:19.173-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:19.230-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:19.287-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:19.287-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:19.287-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:49:19.843-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:19.870-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:19.926-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:19.926-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:19.926-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:49:20.482-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:20.511-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:20.566-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:20.566-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:20.566-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:49:21.122-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:21.151-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:21.205-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:21.205-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:21.205-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:49:21.734-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:21.789-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:21.843-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:21.843-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:21.843-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:49:22.371-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:22.427-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:22.484-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:22.484-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:22.484-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:49:23.042-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:23.069-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:23.125-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:23.125-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:23.125-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:49:23.682-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:23.739-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:23.768-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:23.768-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:23.768-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:49:24.295-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:24.349-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:24.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:24.406-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:24.406-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:49:24.406-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:25.632-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:26.407-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:28.407-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:30.407-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:31.843-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:49:32.407-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:49:32.407-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:34.465-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:34.519-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:34.548-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:34.548-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:34.548-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:49:35.104-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:35.131-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:35.186-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:35.186-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:35.186-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:49:35.716-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:35.771-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:35.827-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:35.827-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:35.827-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:49:36.381-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:36.408-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:36.465-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:36.465-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:36.465-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:49:37.019-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:37.047-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:37.104-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:37.104-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:37.104-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:49:37.660-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:37.714-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:37.741-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:37.741-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:37.741-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:49:38.297-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:38.326-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:38.379-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:38.379-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:38.379-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:49:38.937-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:38.964-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:39.020-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:39.020-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:39.020-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:49:39.576-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:39.603-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:39.659-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:39.659-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:39.659-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:49:40.214-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:40.272-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:40.301-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:40.301-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:40.301-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:49:40.828-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:40.886-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:40.941-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:40.941-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:40.941-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:49:41.497-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:41.524-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:41.580-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:41.580-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:41.580-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:49:41.897-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:41.955-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:41.984-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:41.984-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:49:41.984-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:49:42.107-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:42.163-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:42.217-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:42.217-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:42.217-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:49:42.744-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:42.800-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:42.856-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:42.856-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:42.856-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:49:43.413-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:43.469-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:43.496-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:43.496-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:43.496-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:49:44.051-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:44.104-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:44.132-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:44.132-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:44.132-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:49:44.687-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:44.717-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:44.772-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:44.772-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:44.772-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:49:45.328-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:45.355-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:45.411-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:45.411-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:45.411-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:49:45.411-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:49:47.438-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:47.492-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:47.546-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:47.546-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:47.546-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:49:48.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:48.127-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:48.183-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:48.183-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:48.183-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:49:48.741-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:48.798-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:48.827-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:48.828-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:48.828-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:49:49.385-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:49.414-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:49.470-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:49.470-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:49.470-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:49:49.999-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:50.055-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:50.110-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:50.110-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:50.110-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:49:50.668-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:50.695-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:50.751-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:50.751-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:50.751-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:49:51.278-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:51.334-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:51.388-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:51.388-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:51.388-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:49:51.941-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:51.969-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:52.026-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:52.026-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:52.026-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:49:52.583-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:52.639-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:52.666-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:52.666-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:52.666-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:49:53.222-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:53.278-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:53.305-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:53.305-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:53.305-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:49:53.859-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:49:53.914-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:49:53.943-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:49:53.944-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:49:53.944-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:49:53.944-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:55.632-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:55.944-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:57.944-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:49:59.944-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:01.944-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:02.026-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:50:03.945-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:50:03.945-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:06.001-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:06.030-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:06.087-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:06.087-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:06.087-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:50:06.643-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:06.670-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:06.724-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:06.724-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:06.724-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:50:07.280-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:07.307-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:07.360-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:07.360-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:07.361-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:50:07.888-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:07.945-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:07.999-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:07.999-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:07.999-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:50:08.555-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:08.610-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:08.637-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:08.637-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:08.637-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:50:09.165-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:09.220-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:09.275-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:09.276-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:09.276-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:50:09.831-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:09.858-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:09.915-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:09.916-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:09.916-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:50:10.445-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:10.502-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:10.556-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:10.556-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:10.556-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:50:11.113-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:11.140-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:11.198-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:11.198-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:11.198-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:50:11.754-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:11.781-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:11.834-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:11.834-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:11.834-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:50:12.055-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:12.111-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:12.164-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:12.164-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:50:12.164-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:50:12.362-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:12.417-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:12.475-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:12.475-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:12.475-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:50:13.030-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:13.084-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:13.113-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:13.113-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:13.113-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:50:13.641-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:13.694-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:13.750-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:13.750-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:13.750-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:50:14.279-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:14.333-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:14.386-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:14.386-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:14.386-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:50:14.944-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:14.971-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:15.025-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:15.025-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:15.025-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:50:15.554-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:15.610-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:15.664-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:15.664-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:15.664-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:50:16.193-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:16.251-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:16.304-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:16.304-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:16.304-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:50:16.858-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:16.912-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:16.939-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:16.939-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:16.939-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:50:16.939-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:50:18.997-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:19.024-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:19.079-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:19.080-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:19.080-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:50:19.609-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:19.664-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:19.722-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:19.722-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:19.722-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:50:20.251-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:20.307-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:20.360-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:20.361-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:20.361-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:50:20.918-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:20.975-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:21.002-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:21.002-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:21.003-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:50:21.558-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:21.614-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:21.641-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:21.641-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:21.641-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:50:22.168-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:22.220-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:22.226-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:22.226-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:22.226-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:50:22.782-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:22.809-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:22.864-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:22.864-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:22.864-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:50:23.422-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:23.449-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:23.504-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:23.505-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:23.505-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:50:24.032-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:24.087-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:24.141-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:24.141-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:24.141-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:50:24.697-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:24.751-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:24.778-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:24.778-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:24.778-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:50:25.307-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:25.362-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:25.416-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:25.416-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:25.416-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:50:25.416-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:25.632-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:27.416-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:29.417-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:31.417-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:32.226-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:50:33.417-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:50:33.417-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:35.475-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:35.502-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:35.558-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:35.558-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:35.558-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:50:36.085-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:36.141-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:36.195-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:36.195-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:36.195-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:50:36.752-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:36.808-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:36.835-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:36.835-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:36.835-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:50:37.391-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:37.420-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:37.476-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:37.476-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:37.476-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:50:38.005-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:38.062-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:38.118-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:38.118-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:38.118-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:50:38.674-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:38.701-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:38.754-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:38.755-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:38.755-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:50:39.282-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:39.339-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:39.393-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:39.393-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:39.393-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:50:39.921-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:39.976-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:40.034-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:40.034-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:40.034-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:50:40.589-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:40.643-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:40.672-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:40.672-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:40.672-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:50:41.228-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:41.255-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:41.309-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:41.309-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:41.309-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:50:41.838-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:41.894-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:41.950-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:41.950-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:41.950-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:50:42.282-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:42.339-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:42.368-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:42.368-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:50:42.368-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:50:42.505-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:42.561-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:42.588-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:42.588-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:42.588-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:50:43.117-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:43.173-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:43.229-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:43.229-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:43.229-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:50:43.756-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:43.812-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:43.869-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:43.869-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:43.869-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:50:44.425-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:44.454-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:44.508-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:44.508-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:44.508-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:50:45.064-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:45.119-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:45.146-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:45.146-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:45.146-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:50:45.673-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:45.728-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:45.781-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:45.781-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:45.781-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:50:46.310-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:46.366-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:46.421-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:46.421-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:46.421-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:50:46.421-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:50:48.477-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:48.534-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:48.562-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:48.562-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:48.562-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:50:49.091-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:49.145-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:49.198-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:49.198-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:49.198-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:50:49.728-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:49.785-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:49.839-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:49.839-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:49.839-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:50:50.395-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:50.424-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:50.478-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:50.478-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:50.478-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:50:51.035-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:51.064-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:51.120-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:51.120-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:51.120-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:50:51.649-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:51.705-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:51.758-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:51.758-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:51.758-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:50:52.314-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:52.368-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:52.395-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:52.395-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:52.395-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:50:52.923-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:52.980-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:53.036-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:53.036-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:53.036-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:50:53.593-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:53.649-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:53.676-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:53.676-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:53.676-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:50:54.230-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:54.285-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:54.314-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:54.314-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:54.315-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:50:54.842-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:50:54.900-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:50:54.955-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:50:54.955-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:50:54.955-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:50:54.955-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:55.632-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:56.955-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:50:58.956-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:00.956-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:02.395-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:51:02.956-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:51:02.956-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:05.012-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:05.068-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:05.095-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:05.095-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:05.095-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:51:05.651-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:05.704-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:05.733-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:05.733-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:05.733-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:51:06.291-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:06.320-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:06.373-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:06.373-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:06.373-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:51:06.900-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:06.956-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:07.011-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:07.011-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:07.011-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:51:07.569-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:07.625-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:07.652-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:07.652-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:07.652-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:51:08.206-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:08.233-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:08.290-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:08.290-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:08.290-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:51:08.846-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:08.904-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:08.931-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:08.931-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:08.931-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:51:09.487-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:09.514-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:09.571-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:09.571-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:09.571-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:51:10.127-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:10.183-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:10.210-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:10.210-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:10.210-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:51:10.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:10.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:10.847-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:10.847-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:10.847-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:51:11.374-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:11.432-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:11.489-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:11.489-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:11.489-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:51:12.018-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:12.074-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:12.130-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:12.130-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:12.130-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:51:12.451-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:12.506-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:12.534-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:12.534-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:51:12.534-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:51:12.686-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:12.715-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:12.771-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:12.771-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:12.771-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:51:13.326-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:13.382-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:13.411-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:13.411-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:13.411-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:51:13.967-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:14.020-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:14.049-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:14.049-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:14.049-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:51:14.607-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:14.661-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:14.688-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:14.688-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:14.688-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:51:15.245-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:15.272-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:15.329-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:15.329-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:15.329-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:51:15.858-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:15.914-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:15.970-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:15.970-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:15.970-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:51:15.970-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:51:18.025-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:18.053-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:18.108-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:18.108-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:18.108-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:51:18.664-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:18.722-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:18.749-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:18.749-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:18.749-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:51:19.306-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:19.333-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:19.387-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:19.387-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:19.387-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:51:19.914-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:19.970-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:20.024-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:20.024-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:20.024-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:51:20.581-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:20.635-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:20.664-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:20.664-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:20.664-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:51:21.193-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:21.250-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:21.308-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:21.308-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:21.308-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:51:21.865-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:21.921-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:21.948-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:21.948-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:21.948-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:51:22.504-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:22.558-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:22.563-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:22.563-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:51:22.563-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:51:23.119-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:23.174-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:23.203-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:23.203-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:23.203-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:51:23.761-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:23.816-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:23.843-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:23.843-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:23.843-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:51:24.397-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:24.426-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:24.482-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:24.482-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:24.482-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:51:24.482-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:25.632-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:26.482-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:28.482-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:30.482-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:32.483-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:32.563-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:51:34.483-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:51:34.483-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:36.510-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:36.568-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:36.625-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:36.625-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:36.625-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:51:37.152-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:37.210-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:37.265-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:37.266-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:37.266-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:51:37.821-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:37.875-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:37.902-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:37.902-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:37.902-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:51:38.460-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:38.517-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:38.544-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:38.544-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:38.544-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:51:39.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:39.129-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:39.183-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:39.183-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:39.183-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:51:39.740-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:39.770-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:39.825-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:39.825-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:39.825-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:51:40.381-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:40.410-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:40.464-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:40.464-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:40.464-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:51:40.991-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:41.047-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:41.105-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:41.105-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:41.105-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:51:41.658-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:41.686-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:41.744-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:41.744-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:41.744-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:51:42.298-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:42.355-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:42.382-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:42.382-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:42.382-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:51:42.620-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:42.678-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:42.707-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:42.707-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:51:42.707-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:51:42.936-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:42.965-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:43.022-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:43.023-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:43.023-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:51:43.550-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:43.607-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:43.663-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:43.663-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:43.663-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:51:44.220-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:44.274-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:44.302-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:44.302-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:44.302-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:51:44.858-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:44.887-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:44.943-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:44.943-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:44.943-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:51:45.472-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:45.527-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:45.581-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:45.581-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:45.581-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:51:46.108-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:46.164-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:46.218-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:46.218-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:46.218-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:51:46.775-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:46.829-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:46.858-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:46.858-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:46.858-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:51:47.387-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:47.443-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:47.500-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:47.500-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:47.500-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:51:47.500-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:51:49.527-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:49.581-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:49.637-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:49.637-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:49.637-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:51:50.164-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:50.220-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:50.273-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:50.273-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:50.273-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:51:50.802-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:50.858-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:50.914-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:50.914-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:50.914-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:51:51.443-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:51.500-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:51.558-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:51.558-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:51.558-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:51:52.085-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:52.143-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:52.198-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:52.198-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:52.198-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:51:52.725-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:52.761-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:52.781-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:52.782-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:52.782-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:51:53.309-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:53.366-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:53.422-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:53.422-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:53.422-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:51:53.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:54.035-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:54.062-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:54.062-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:54.062-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:51:54.619-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:54.649-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:54.706-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:54.706-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:54.706-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:51:55.262-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:55.319-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:55.346-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:55.346-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:51:55.346-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:51:55.688-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:51:55.742-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:51:55.769-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:51:55.769-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:51:55.769-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:51:55.769-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:55.846-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:57.846-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:51:59.847-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:01.847-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:02.782-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:52:03.847-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:52:03.847-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:05.903-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:05.956-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:05.984-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:05.984-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:05.984-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:52:06.541-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:06.597-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:06.626-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:06.626-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:06.626-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:52:07.155-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:07.211-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:07.264-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:07.265-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:07.265-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:52:07.792-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:07.847-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:07.905-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:07.905-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:07.905-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:52:08.462-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:08.518-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:08.545-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:08.545-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:08.545-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:52:09.101-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:09.130-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:09.186-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:09.186-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:09.186-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:52:09.739-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:09.767-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:09.822-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:09.822-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:09.822-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:52:10.380-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:10.407-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:10.465-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:10.465-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:10.465-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:52:11.018-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:11.072-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:11.099-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:11.099-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:11.099-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:52:11.657-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:11.686-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:11.741-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:11.741-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:11.741-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:52:12.295-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:12.351-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:12.378-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:12.378-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:12.378-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:52:12.811-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:12.868-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:12.926-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:12.926-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:52:12.926-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:52:13.482-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:13.509-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:13.564-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:13.564-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:13.564-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:52:14.094-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:14.147-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:14.201-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:14.201-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:14.201-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:52:14.730-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:14.786-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:14.841-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:14.841-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:14.841-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:52:15.395-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:15.451-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:15.480-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:15.480-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:15.480-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:52:16.034-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:16.061-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:16.118-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:16.118-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:16.118-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:52:16.672-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:16.726-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:16.753-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:16.753-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:16.753-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:52:16.753-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:52:18.809-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:18.836-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:18.893-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:18.893-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:18.893-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:52:19.449-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:19.506-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:19.534-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:19.534-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:19.534-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:52:20.089-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:20.145-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:20.172-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:20.172-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:20.172-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:52:20.702-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:20.757-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:20.813-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:20.813-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:20.813-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:52:21.340-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:21.394-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:21.449-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:21.449-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:21.449-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:52:22.003-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:22.059-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:22.086-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:22.086-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:22.086-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:52:22.640-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:22.697-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:22.726-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:22.726-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:22.726-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:52:22.981-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:23.008-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:23.064-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:23.064-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:52:23.064-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:52:23.280-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:23.334-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:23.361-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:23.361-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:23.361-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:52:23.888-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:23.944-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:23.998-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:23.998-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:23.998-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:52:24.527-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:24.581-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:24.636-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:24.636-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:24.636-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:52:25.192-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:25.249-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:25.278-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:25.278-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:25.278-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:52:25.278-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:25.769-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:27.278-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:29.279-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:31.279-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:33.064-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:52:33.279-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:52:33.279-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:35.335-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:35.388-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:35.418-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:35.418-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:35.418-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:52:35.971-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:35.999-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:36.054-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:36.054-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:36.054-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:52:36.608-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:36.637-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:36.693-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:36.693-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:36.693-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:52:37.220-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:37.274-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:37.328-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:37.328-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:37.328-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:52:37.883-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:37.912-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:37.968-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:37.968-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:37.968-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:52:38.523-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:38.579-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:38.606-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:38.606-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:38.606-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:52:39.135-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:39.189-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:39.245-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:39.245-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:39.245-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:52:39.772-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:39.830-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:39.887-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:39.887-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:39.887-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:52:40.416-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:40.472-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:40.527-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:40.527-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:40.527-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:52:41.057-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:41.110-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:41.166-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:41.166-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:41.166-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:52:41.695-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:41.751-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:41.808-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:41.808-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:41.808-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:52:42.335-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:42.393-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:42.450-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:42.450-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:42.450-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:52:43.004-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:43.060-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:43.089-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:43.089-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:43.089-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:52:43.646-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:43.675-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:43.733-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:43.733-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:43.733-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:52:44.260-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:44.316-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:44.371-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:44.371-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:44.371-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:52:44.927-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:44.956-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:45.010-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:45.010-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:45.010-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:52:45.539-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:45.593-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:45.646-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:45.646-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:45.646-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:52:46.174-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:46.229-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:46.283-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:46.283-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:46.283-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:52:46.283-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:52:48.339-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:48.368-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:48.423-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:48.424-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:48.424-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:52:48.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:49.031-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:49.058-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:49.058-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:49.058-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:52:49.614-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:49.669-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:49.697-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:49.697-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:49.697-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:52:50.252-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:50.280-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:50.335-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:50.335-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:50.335-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:52:50.863-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:50.918-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:50.974-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:50.974-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:50.974-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:52:51.531-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:51.560-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:51.614-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:51.614-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:51.614-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:52:52.168-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:52.223-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:52.253-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:52.253-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:52.253-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:52:52.810-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:52.837-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:52.891-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:52.891-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:52.891-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:52:53.118-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:53.174-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:53.229-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:53.229-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:52:53.229-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:52:53.449-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:53.504-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:53.533-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:53.533-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:53.533-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:52:54.061-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:54.116-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:54.172-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:54.172-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:54.172-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:52:54.728-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:52:54.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:52:54.810-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:52:54.811-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:52:54.811-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:52:54.811-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:55.769-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:56.811-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:52:58.811-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:00.811-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:02.811-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:03.229-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:53:04.811-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:53:04.811-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:06.839-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:06.893-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:06.946-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:06.946-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:06.946-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:53:07.502-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:07.529-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:07.585-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:07.585-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:07.585-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:53:08.141-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:08.198-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:08.225-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:08.225-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:08.225-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:53:08.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:08.810-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:08.867-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:08.867-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:08.867-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:53:09.397-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:09.450-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:09.508-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:09.508-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:09.508-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:53:10.064-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:10.093-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:10.148-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:10.148-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:10.148-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:53:10.704-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:10.733-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:10.791-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:10.791-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:10.791-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:53:11.318-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:11.371-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:11.427-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:11.427-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:11.427-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:53:11.981-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:12.008-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:12.063-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:12.063-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:12.063-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:53:12.617-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:12.675-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:12.704-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:12.704-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:12.704-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:53:13.233-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:13.287-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:13.289-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:13.289-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:13.289-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:53:13.842-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:13.872-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:13.929-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:13.929-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:13.929-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:53:14.456-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:14.510-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:14.566-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:14.566-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:14.566-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:53:15.121-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:15.175-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:15.204-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:15.204-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:15.204-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:53:15.760-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:15.817-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:15.846-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:15.846-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:15.846-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:53:16.400-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:16.458-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:16.487-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:16.487-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:16.487-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:53:17.042-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:17.096-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:17.123-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:17.124-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:17.124-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:53:17.653-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:17.708-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:17.764-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:17.764-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:17.764-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:53:17.764-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:53:19.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:19.847-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:19.902-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:19.902-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:19.902-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:53:20.458-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:20.487-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:20.543-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:20.543-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:20.543-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:53:21.098-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:21.125-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:21.181-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:21.181-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:21.181-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:53:21.738-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:21.794-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:21.823-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:21.823-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:21.823-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:53:22.352-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:22.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:22.462-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:22.462-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:22.462-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:53:23.019-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:23.073-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:23.100-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:23.100-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:23.100-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:53:23.344-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:23.400-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:23.427-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:23.427-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:53:23.427-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:53:23.656-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:23.711-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:23.739-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:23.739-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:23.739-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:53:24.294-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:24.322-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:24.377-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:24.377-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:24.377-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:53:24.935-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:24.990-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:25.017-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:25.018-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:25.018-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:53:25.545-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:25.602-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:25.658-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:25.658-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:25.658-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:53:25.825-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:25.879-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:25.908-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:25.908-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:53:25.908-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:53:25.908-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:26.158-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:28.158-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:30.158-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:32.158-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:33.427-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:53:34.158-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:53:34.158-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:36.186-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:36.241-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:36.297-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:36.297-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:36.297-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:53:36.851-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:36.906-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:36.933-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:36.933-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:36.934-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:53:37.489-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:37.518-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:37.572-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:37.572-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:37.572-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:53:38.128-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:38.157-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:38.210-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:38.210-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:38.210-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:53:38.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:38.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:38.851-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:38.851-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:38.851-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:53:39.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:39.434-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:39.491-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:39.491-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:39.491-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:53:40.047-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:40.103-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:40.132-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:40.132-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:40.132-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:53:40.659-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:40.715-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:40.770-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:40.770-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:40.770-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:53:41.298-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:41.353-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:41.409-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:41.409-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:41.409-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:53:41.938-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:41.993-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:42.051-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:42.051-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:42.051-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:53:42.607-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:42.636-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:42.693-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:42.693-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:42.693-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:53:43.247-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:43.274-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:43.330-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:43.330-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:43.330-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:53:43.483-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:43.539-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:43.568-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:43.568-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:53:43.568-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:53:43.857-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:43.913-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:43.967-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:43.967-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:43.967-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:53:44.522-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:44.550-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:44.603-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:44.603-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:44.603-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:53:45.157-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:45.211-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:45.238-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:45.238-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:45.238-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:53:45.766-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:45.819-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:45.873-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:45.873-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:45.873-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:53:46.402-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:46.460-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:46.514-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:46.514-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:46.514-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:53:47.071-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:47.098-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:47.154-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:47.154-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:47.154-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:53:47.154-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:53:49.183-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:49.239-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:49.294-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:49.294-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:49.294-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:53:49.852-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:49.909-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:49.936-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:49.936-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:49.936-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:53:50.466-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:50.521-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:50.579-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:50.579-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:50.579-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:53:51.108-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:51.163-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:51.217-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:51.217-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:51.217-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:53:51.773-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:51.800-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:51.856-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:51.856-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:51.856-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:53:52.383-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:52.437-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:52.490-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:52.490-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:52.490-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:53:53.018-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:53.073-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:53.131-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:53.131-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:53.131-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:53:53.595-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:53.651-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:53.686-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:53.687-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:53.687-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:53:54.240-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:54.267-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:54.325-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:54.325-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:54.325-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:53:54.854-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:54.908-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:54.963-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:54.963-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:54.963-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:53:55.519-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:53:55.576-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:53:55.604-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:53:55.604-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:53:55.604-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:53:55.604-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:55.908-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:57.604-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:53:59.604-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:01.604-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:03.604-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:03.687-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:54:05.605-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:54:05.605-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:07.660-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:07.714-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:07.741-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:07.741-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:07.741-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:54:08.295-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:08.322-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:08.376-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:08.376-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:08.376-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:54:08.904-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:08.959-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:09.015-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:09.015-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:09.015-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:54:09.572-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:09.601-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:09.657-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:09.657-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:09.657-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:54:10.211-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:10.240-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:10.295-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:10.295-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:10.296-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:54:10.851-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:10.880-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:10.936-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:10.936-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:10.936-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:54:11.491-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:11.547-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:11.576-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:11.576-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:11.576-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:54:12.105-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:12.161-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:12.215-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:12.215-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:12.215-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:54:12.769-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:12.824-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:12.851-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:12.851-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:12.852-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:54:13.381-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:13.434-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:13.492-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:13.492-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:13.492-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:54:13.742-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:13.771-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:13.827-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:13.827-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:54:13.827-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:54:14.047-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:14.105-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:14.134-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:14.134-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:14.134-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:54:14.663-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:14.717-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:14.775-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:14.775-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:14.775-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:54:15.302-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:15.358-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:15.413-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:15.413-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:15.413-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:54:15.967-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:16.023-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:16.052-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:16.052-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:16.052-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:54:16.581-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:16.639-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:16.696-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:16.696-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:16.696-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:54:17.252-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:17.307-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:17.335-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:17.335-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:17.335-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:54:17.888-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:17.944-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:17.973-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:17.973-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:17.973-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:54:18.527-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:18.556-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:18.614-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:18.614-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:18.614-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:54:18.614-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:54:20.641-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:20.698-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:20.756-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:20.756-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:20.756-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:54:21.283-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:21.337-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:21.394-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:21.394-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:21.394-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:54:21.923-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:21.979-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:22.033-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:22.033-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:22.033-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:54:22.560-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:22.614-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:22.671-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:22.671-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:22.671-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:54:23.227-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:23.282-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:23.312-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:23.312-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:23.312-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:54:23.841-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:23.883-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:23.896-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:23.896-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:23.896-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:54:24.426-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:24.479-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:24.537-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:24.537-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:24.537-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:54:25.092-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:25.121-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:25.177-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:25.177-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:25.177-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:54:25.731-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:25.760-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:25.814-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:25.814-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:25.814-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:54:25.964-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:25.991-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:26.045-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:26.045-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:54:26.045-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:54:26.343-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:26.397-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:26.454-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:26.454-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:26.454-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:54:26.454-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:26.545-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:28.455-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:30.455-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:32.455-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:33.897-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:54:34.455-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:54:34.455-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:36.482-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:36.536-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:36.591-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:36.592-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:36.592-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:54:37.119-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:37.175-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:37.230-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:37.230-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:37.230-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:54:37.759-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:37.817-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:37.872-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:37.873-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:37.873-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:54:38.430-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:38.486-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:38.515-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:38.515-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:38.515-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:54:39.069-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:39.098-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:39.153-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:39.153-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:39.153-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:54:39.681-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:39.735-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:39.788-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:39.788-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:39.788-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:54:40.316-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:40.373-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:40.429-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:40.429-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:40.429-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:54:40.983-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:41.036-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:41.065-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:41.065-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:41.065-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:54:41.623-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:41.677-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:41.706-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:41.706-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:41.706-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:54:42.233-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:42.289-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:42.346-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:42.346-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:42.346-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:54:42.873-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:42.929-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:42.985-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:42.985-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:42.985-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:54:43.539-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:43.594-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:43.623-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:43.623-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:43.623-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:54:43.924-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:43.980-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:44.035-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:44.035-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:54:44.035-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:54:44.181-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:44.210-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:44.265-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:44.265-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:44.265-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:54:44.821-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:44.875-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:44.902-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:44.902-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:44.902-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:54:45.458-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:45.487-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:45.543-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:45.543-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:45.543-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:54:46.096-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:46.152-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:46.179-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:46.179-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:46.179-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:54:46.735-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:46.762-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:46.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:46.820-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:46.820-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:54:47.377-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:47.407-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:47.460-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:47.460-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:47.460-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:54:47.460-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:54:49.514-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:49.541-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:49.597-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:49.597-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:49.597-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:54:50.126-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:50.184-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:50.241-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:50.241-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:50.241-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:54:50.770-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:50.828-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:50.883-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:50.883-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:50.883-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:54:51.439-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:51.466-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:51.524-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:51.524-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:51.524-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:54:52.053-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:52.109-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:52.164-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:52.164-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:52.164-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:54:52.692-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:52.747-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:52.803-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:52.803-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:52.803-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:54:53.357-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:53.410-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:53.440-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:53.440-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:53.440-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:54:53.995-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:54.049-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:54.062-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:54.063-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:54:54.063-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:54:54.620-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:54.649-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:54.705-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:54.705-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:54.705-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:54:55.261-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:55.318-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:55.345-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:55.345-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:55.345-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:54:55.901-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:54:55.957-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:54:55.986-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:54:55.986-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:54:55.986-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:54:55.986-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:56.545-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:57.986-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:54:59.986-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:01.986-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:03.986-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:04.063-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:55:05.987-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:55:05.987-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:08.014-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:08.072-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:08.127-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:08.127-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:08.127-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:55:08.683-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:08.710-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:08.764-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:08.764-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:08.764-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:55:09.320-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:09.373-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:09.402-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:09.403-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:09.403-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:55:09.956-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:10.014-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:10.041-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:10.041-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:10.041-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:55:10.570-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:10.626-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:10.681-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:10.682-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:10.682-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:55:11.237-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:11.293-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:11.320-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:11.320-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:11.320-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:55:11.848-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:11.901-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:11.957-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:11.957-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:11.957-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:55:12.486-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:12.542-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:12.599-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:12.599-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:12.599-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:55:13.155-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:13.182-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:13.238-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:13.238-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:13.238-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:55:13.765-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:13.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:13.876-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:13.876-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:13.876-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:55:14.090-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:14.144-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:14.199-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:14.199-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:55:14.199-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:55:14.405-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:14.459-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:14.515-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:14.515-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:14.515-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:55:15.070-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:15.099-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:15.157-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:15.157-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:15.157-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:55:15.713-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:15.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:15.794-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:15.794-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:15.794-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:55:16.351-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:16.378-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:16.434-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:16.434-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:16.434-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:55:16.990-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:17.047-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:17.076-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:17.076-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:17.076-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:55:17.605-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:17.661-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:17.716-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:17.717-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:17.717-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:55:18.244-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:18.299-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:18.355-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:18.355-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:18.355-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:55:18.911-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:18.940-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:18.997-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:18.997-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:18.997-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:55:18.997-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:55:21.055-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:21.112-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:21.140-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:21.140-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:21.140-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:55:21.697-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:21.751-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:21.780-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:21.780-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:21.780-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:55:22.309-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:22.365-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:22.420-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:22.420-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:22.420-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:55:22.974-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:23.030-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:23.059-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:23.059-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:23.059-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:55:23.616-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:23.646-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:23.703-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:23.703-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:23.703-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:55:24.255-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:24.261-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:24.282-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:24.282-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:55:24.282-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:55:24.810-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:24.867-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:24.921-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:24.921-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:24.921-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:55:25.475-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:25.502-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:25.559-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:25.560-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:25.560-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:55:26.089-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:26.146-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:26.204-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:26.204-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:26.204-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:55:26.599-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:26.656-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:26.685-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:26.685-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:55:26.685-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:55:26.757-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:26.787-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:26.844-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:26.844-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:26.844-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:55:26.844-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:27.186-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:28.844-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:30.844-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:32.844-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:34.282-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:55:34.844-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:55:34.844-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:36.900-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:36.956-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:36.983-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:36.983-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:36.983-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:55:37.537-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:37.592-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:37.620-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:37.620-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:37.620-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:55:38.177-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:38.235-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:38.264-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:38.264-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:38.264-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:55:38.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:38.873-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:38.901-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:38.901-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:38.901-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:55:39.456-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:39.485-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:39.543-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:39.543-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:39.543-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:55:40.070-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:40.126-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:40.183-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:40.183-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:40.183-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:55:40.739-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:40.766-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:40.823-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:40.824-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:40.824-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:55:41.379-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:41.437-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:41.464-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:41.464-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:41.464-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:55:41.993-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:42.049-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:42.104-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:42.105-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:42.105-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:55:42.662-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:42.691-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:42.749-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:42.749-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:42.749-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:55:43.278-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:43.333-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:43.391-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:43.391-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:43.391-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:55:43.920-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:43.976-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:44.033-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:44.033-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:44.033-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:55:44.336-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:44.365-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:44.421-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:44.421-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:55:44.421-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:55:44.587-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:44.614-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:44.668-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:44.668-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:44.668-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:55:45.197-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:45.253-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:45.308-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:45.308-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:45.308-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:55:45.836-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:45.893-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:45.949-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:45.949-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:45.949-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:55:46.504-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:46.562-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:46.591-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:46.591-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:46.591-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:55:47.145-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:47.200-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:47.230-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:47.230-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:47.230-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:55:47.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:47.811-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:47.866-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:47.866-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:47.866-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:55:47.866-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:55:49.924-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:49.980-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:50.009-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:50.009-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:50.009-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:55:50.536-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:50.592-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:50.649-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:50.649-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:50.649-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:55:51.178-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:51.234-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:51.289-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:51.289-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:51.289-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:55:51.819-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:51.876-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:51.930-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:51.930-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:51.930-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:55:52.457-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:52.511-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:52.567-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:52.567-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:52.567-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:55:53.120-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:53.149-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:53.205-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:53.205-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:53.205-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:55:53.761-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:53.788-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:53.844-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:53.844-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:53.844-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:55:54.397-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:54.426-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:54.478-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:54.478-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:55:54.479-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:55:55.006-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:55.060-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:55.113-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:55.113-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:55.113-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:55:55.643-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:55.698-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:55.752-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:55.752-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:55.752-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:55:56.308-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:55:56.337-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:55:56.391-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:55:56.391-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:55:56.391-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:55:56.391-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:57.186-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:55:58.391-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:00.391-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:02.391-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:04.391-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:04.479-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:56:06.391-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:56:06.391-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:08.447-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:08.505-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:08.532-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:08.532-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:08.532-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:56:09.059-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:09.115-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:09.171-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:09.171-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:09.171-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:56:09.728-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:09.782-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:09.811-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:09.811-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:09.811-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:56:10.365-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:10.419-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:10.446-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:10.446-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:10.446-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:56:10.975-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:11.031-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:11.088-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:11.088-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:11.088-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:56:11.642-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:11.698-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:11.727-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:11.727-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:11.727-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:56:12.281-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:12.336-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:12.364-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:12.364-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:12.364-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:56:12.920-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:12.973-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:13.001-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:13.001-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:13.001-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:56:13.530-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:13.587-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:13.641-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:13.641-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:13.641-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:56:14.197-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:14.224-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:14.280-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:14.280-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:14.280-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:56:14.536-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:14.592-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:14.621-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:14.621-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:56:14.621-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:56:14.835-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:14.891-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:14.920-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:14.920-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:14.920-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:56:15.449-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:15.503-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:15.557-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:15.557-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:15.557-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:56:16.114-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:16.170-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:16.197-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:16.197-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:16.197-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:56:16.726-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:16.780-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:16.836-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:16.836-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:16.836-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:56:17.391-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:17.418-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:17.474-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:17.474-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:17.474-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:56:18.028-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:18.085-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:18.113-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:18.113-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:18.113-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:56:18.667-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:18.694-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:18.750-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:18.750-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:18.750-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:56:19.305-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:19.334-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:19.392-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:19.392-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:19.392-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:56:19.392-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:56:21.448-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:21.503-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:21.532-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:21.532-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:21.532-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:56:22.086-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:22.144-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:22.173-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:22.173-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:22.173-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:56:22.728-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:22.758-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:22.811-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:22.811-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:22.811-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:56:23.367-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:23.396-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:23.450-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:23.450-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:23.450-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:56:24.005-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:24.063-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:24.092-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:24.092-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:24.092-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:56:24.648-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:24.650-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:24.701-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:24.701-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:24.701-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:56:25.229-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:25.284-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:25.340-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:25.340-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:25.340-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:56:25.895-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:25.951-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:25.978-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:25.979-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:25.979-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:56:26.536-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:26.563-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:26.619-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:26.619-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:26.619-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:56:27.176-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:27.215-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:27.234-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:27.234-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:27.234-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:56:27.788-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:27.790-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:27.817-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:27.817-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:27.817-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:56:27.817-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:27.817-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:29.817-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:31.817-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:33.817-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:34.701-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:56:35.817-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:56:35.817-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:37.845-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:37.898-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:37.954-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:37.954-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:37.954-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:56:38.483-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:38.537-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:38.592-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:38.592-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:38.592-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:56:39.148-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:39.205-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:39.233-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:39.233-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:39.233-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:56:39.762-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:39.818-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:39.873-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:39.873-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:39.873-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:56:40.429-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:40.458-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:40.513-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:40.513-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:40.513-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:56:41.043-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:41.096-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:41.154-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:41.154-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:41.154-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:56:41.683-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:41.739-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:41.793-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:41.793-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:41.793-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:56:42.346-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:42.374-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:42.427-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:42.427-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:42.427-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:56:42.983-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:43.010-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:43.066-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:43.066-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:43.066-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:56:43.620-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:43.677-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:43.704-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:43.704-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:43.704-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:56:44.260-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:44.316-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:44.345-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:44.345-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:44.345-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:56:44.731-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:44.789-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:44.844-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:44.844-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:56:44.844-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:56:44.900-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:44.956-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:44.983-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:44.983-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:44.983-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:56:45.537-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:45.593-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:45.620-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:45.620-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:45.620-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:56:46.149-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:46.205-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:46.260-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:46.260-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:46.260-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:56:46.788-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:46.845-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:46.903-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:46.903-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:46.903-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:56:47.456-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:47.514-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:47.543-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:47.543-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:47.543-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:56:48.099-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:48.126-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:48.183-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:48.183-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:48.183-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:56:48.713-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:48.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:48.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:48.820-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:48.820-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:56:48.820-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:56:50.874-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:50.930-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:50.959-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:50.959-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:50.959-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:56:51.488-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:51.543-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:51.601-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:51.601-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:51.601-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:56:52.157-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:52.212-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:52.239-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:52.239-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:52.239-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:56:52.769-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:52.822-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:52.876-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:52.876-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:52.876-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:56:53.434-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:53.463-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:53.518-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:53.518-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:53.518-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:56:54.074-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:54.129-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:54.157-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:54.157-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:54.157-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:56:54.684-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:54.742-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:54.797-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:54.797-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:54.797-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:56:54.872-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:54.925-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:54.983-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:54.983-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:56:54.983-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:56:55.351-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:55.378-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:55.434-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:55.434-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:55.434-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:56:55.988-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:56.044-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:56.073-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:56.073-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:56.073-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:56:56.629-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:56:56.682-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:56:56.710-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:56:56.710-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:56:56.710-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:56:56.710-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:57.817-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:56:58.710-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:00.710-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:02.710-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:04.710-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:04.983-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:57:06.711-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:57:06.711-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:08.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:08.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:08.851-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:08.851-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:08.851-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:57:09.407-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:09.436-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:09.493-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:09.493-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:09.493-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:57:10.022-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:10.078-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:10.134-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:10.134-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:10.134-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:57:10.691-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:10.720-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:10.776-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:10.776-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:10.776-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:57:11.303-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:11.359-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:11.416-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:11.416-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:11.416-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:57:11.943-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:11.997-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:12.053-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:12.053-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:12.053-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:57:12.582-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:12.637-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:12.691-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:12.691-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:12.691-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:57:13.247-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:13.274-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:13.330-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:13.330-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:13.330-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:57:13.857-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:13.914-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:13.970-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:13.970-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:13.970-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:57:14.526-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:14.580-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:14.609-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:14.609-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:14.609-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:57:15.037-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:15.064-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:15.118-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:15.118-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:57:15.118-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:57:15.672-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:15.729-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:15.758-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:15.758-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:15.758-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:57:16.314-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:16.341-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:16.395-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:16.395-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:16.395-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:57:16.949-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:17.003-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:17.032-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:17.032-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:17.032-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:57:17.587-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:17.643-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:17.671-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:17.671-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:17.671-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:57:18.226-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:18.282-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:18.309-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:18.309-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:18.309-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:57:18.837-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:18.894-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:18.950-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:18.950-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:18.950-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:57:19.505-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:19.559-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:19.586-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:19.586-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:19.586-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:57:19.586-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:57:21.640-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:21.669-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:21.727-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:21.727-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:21.727-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:57:22.284-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:22.313-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:22.367-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:22.367-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:22.367-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:57:22.923-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:22.980-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:23.009-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:23.009-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:23.009-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:57:23.538-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:23.594-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:23.648-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:23.648-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:23.648-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:57:24.177-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:24.234-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:24.290-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:24.290-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:24.290-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:57:24.847-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:24.903-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:24.932-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:24.932-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:24.932-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:57:25.173-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:25.201-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:25.258-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:25.258-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:57:25.258-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:57:25.490-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:25.545-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:25.572-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:25.572-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:25.572-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:57:26.126-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:26.184-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:26.213-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:26.213-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:26.213-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:57:26.742-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:26.799-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:26.855-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:26.855-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:26.855-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:57:27.382-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:27.436-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:27.490-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:27.490-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:27.490-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:57:27.844-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:27.902-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:27.959-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:27.959-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:57:27.959-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:57:27.959-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:27.990-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:29.990-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:31.990-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:33.990-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:35.258-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:57:35.991-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:57:35.991-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:38.044-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:38.072-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:38.125-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:38.125-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:38.125-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:57:38.653-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:38.708-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:38.764-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:38.764-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:38.764-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:57:39.318-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:39.345-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:39.399-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:39.399-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:39.399-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:57:39.954-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:39.984-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:40.039-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:40.039-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:40.039-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:57:40.595-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:40.650-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:40.678-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:40.678-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:40.678-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:57:41.231-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:41.261-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:41.318-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:41.318-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:41.318-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:57:41.847-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:41.901-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:41.958-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:41.958-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:41.958-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:57:42.516-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:42.573-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:42.600-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:42.600-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:42.600-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:57:43.156-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:43.214-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:43.243-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:43.243-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:43.243-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:57:43.800-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:43.828-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:43.883-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:43.883-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:43.883-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:57:44.441-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:44.470-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:44.524-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:44.524-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:44.524-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:57:45.078-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:45.105-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:45.161-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:45.161-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:45.161-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:57:45.312-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:45.339-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:45.393-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:45.393-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:57:45.393-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:57:45.716-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:45.744-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:45.799-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:45.799-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:45.799-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:57:46.353-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:46.410-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:46.438-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:46.438-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:46.438-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:57:46.991-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:47.047-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:47.074-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:47.074-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:47.074-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:57:47.602-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:47.657-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:47.711-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:47.711-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:47.711-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:57:48.240-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:48.296-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:48.351-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:48.351-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:48.351-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:57:48.880-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:48.934-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:48.988-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:48.988-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:48.988-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:57:48.988-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:57:51.046-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:51.075-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:51.132-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:51.132-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:51.132-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:57:51.688-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:51.742-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:51.769-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:51.769-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:51.769-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:57:52.323-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:52.350-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:52.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:52.406-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:52.406-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:57:52.961-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:53.015-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:53.042-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:53.042-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:53.042-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:57:53.600-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:53.627-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:53.681-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:53.681-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:53.681-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:57:54.210-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:54.264-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:54.321-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:54.321-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:54.321-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:57:54.875-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:54.904-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:54.961-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:54.962-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:54.962-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:57:55.451-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:55.478-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:55.515-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:55.515-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:55.515-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:57:56.045-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:56.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:56.156-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:56.156-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:56.156-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:57:56.710-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:56.763-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:56.792-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:56.792-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:56.792-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:57:57.346-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:57:57.374-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:57:57.427-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:57:57.427-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:57:57.427-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:57:57.427-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:57.960-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:57:59.428-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:01.428-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:03.428-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:05.428-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:05.516-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:58:07.428-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:58:07.428-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:09.486-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:09.541-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:09.571-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:09.571-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:09.571-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:58:10.100-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:10.157-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:10.213-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:10.213-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:10.213-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:58:10.768-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:10.796-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:10.851-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:10.851-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:10.851-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:58:11.381-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:11.438-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:11.494-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:11.494-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:11.494-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:58:12.049-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:12.103-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:12.130-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:12.131-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:12.131-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:58:12.684-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:12.740-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:12.767-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:12.767-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:12.767-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:58:13.296-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:13.350-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:13.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:13.406-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:13.406-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:58:13.935-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:13.989-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:14.044-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:14.044-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:14.044-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:58:14.600-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:14.656-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:14.685-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:14.685-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:14.685-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:58:15.239-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:15.266-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:15.322-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:15.322-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:15.322-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:58:15.569-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:15.627-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:15.654-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:15.654-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:58:15.654-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:58:15.851-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:15.906-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:15.962-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:15.962-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:15.962-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:58:16.489-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:16.547-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:16.604-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:16.604-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:16.604-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:58:17.162-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:17.189-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:17.246-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:17.246-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:17.246-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:58:17.775-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:17.831-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:17.887-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:17.887-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:17.887-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:58:18.414-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:18.470-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:18.525-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:18.525-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:18.525-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:58:19.079-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:19.106-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:19.164-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:19.164-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:19.164-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:58:19.691-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:19.747-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:19.804-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:19.804-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:19.804-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:58:20.360-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:20.417-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:20.444-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:20.445-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:20.445-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:58:20.445-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:58:22.502-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:22.529-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:22.585-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:22.585-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:22.585-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:58:23.114-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:23.168-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:23.224-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:23.224-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:23.224-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:58:23.753-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:23.806-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:23.860-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:23.860-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:23.860-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:58:24.416-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:24.443-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:24.499-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:24.499-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:24.499-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:58:25.054-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:25.084-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:25.139-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:25.139-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:25.139-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:58:25.666-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:25.710-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:25.722-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:25.722-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:25.722-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:58:26.278-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:26.333-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:26.361-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:26.361-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:26.361-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:58:26.916-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:26.972-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:27.001-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:27.001-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:27.001-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:58:27.557-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:27.586-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:27.641-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:27.641-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:27.641-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:58:28.017-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:28.046-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:28.102-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:28.102-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:58:28.102-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:58:28.199-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:28.226-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:28.282-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:28.282-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:28.282-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:58:28.282-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:28.602-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:30.282-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:32.282-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:34.282-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:35.722-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:58:36.282-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:58:36.282-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:38.338-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:38.367-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:38.423-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:38.423-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:38.423-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:58:38.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:39.006-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:39.061-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:39.061-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:39.061-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:58:39.617-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:39.672-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:39.700-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:39.700-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:39.700-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:58:40.257-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:40.286-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:40.342-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:40.342-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:40.342-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:58:40.871-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:40.925-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:40.982-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:40.983-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:40.983-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:58:41.536-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:41.564-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:41.619-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:41.619-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:41.619-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:58:42.175-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:42.230-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:42.260-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:42.260-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:42.260-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:58:42.815-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:42.873-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:42.902-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:42.902-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:42.902-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:58:43.456-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:43.511-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:43.540-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:43.540-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:43.540-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:58:44.070-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:44.124-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:44.179-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:44.179-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:44.179-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:58:44.735-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:44.762-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:44.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:44.820-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:44.820-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:58:45.349-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:45.403-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:45.456-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:45.456-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:45.456-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:58:45.776-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:45.831-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:45.861-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:45.861-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:58:45.861-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:58:46.012-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:46.068-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:46.095-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:46.095-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:46.095-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:58:46.649-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:46.704-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:46.731-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:46.731-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:46.731-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:58:47.259-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:47.314-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:47.372-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:47.372-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:47.372-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:58:47.926-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:47.979-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:48.007-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:48.007-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:48.007-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:58:48.562-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:48.618-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:48.647-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:48.647-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:48.647-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:58:49.203-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:49.232-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:49.289-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:49.289-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:49.289-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:58:49.289-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:58:51.347-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:51.376-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:51.431-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:51.431-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:51.431-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:58:51.987-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:52.041-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:52.068-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:52.068-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:52.068-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:58:52.625-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:52.654-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:52.708-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:52.708-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:52.708-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:58:53.262-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:53.319-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:53.349-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:53.349-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:53.349-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:58:53.902-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:53.960-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:53.987-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:53.987-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:53.987-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:58:54.516-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:54.574-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:54.631-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:54.631-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:54.631-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:58:55.189-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:55.218-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:55.274-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:55.274-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:55.274-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:58:55.828-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:55.883-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:55.888-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:55.888-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:58:55.888-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:58:56.444-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:56.499-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:56.528-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:56.528-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:56.528-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:58:57.086-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:57.141-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:57.170-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:57.171-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:57.171-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:58:57.726-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:58:57.782-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:58:57.809-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:58:57.809-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:58:57.809-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:58:57.809-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:58.602-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:58:59.809-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:01.809-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:03.810-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:05.810-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:05.888-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:59:07.810-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:59:07.810-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:09.866-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:09.893-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:09.949-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:09.949-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:09.949-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:59:10.502-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:10.530-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:10.583-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:10.583-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:10.583-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:59:11.139-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:11.193-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:11.222-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:11.222-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:11.222-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:59:11.779-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:11.837-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:11.866-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:11.866-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:11.866-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:59:12.395-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:12.451-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:12.506-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:12.506-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:12.506-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:59:13.062-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:13.089-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:13.147-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:13.147-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:13.147-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:59:13.700-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:13.728-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:13.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:13.783-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:13.783-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:59:14.310-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:14.366-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:14.424-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:14.424-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:14.424-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:59:14.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:15.031-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:15.060-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:15.060-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:15.060-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:59:15.614-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:15.643-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:15.700-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:15.700-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:15.700-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:59:15.944-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:15.999-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:16.026-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:16.027-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:59:16.027-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:59:16.228-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:16.285-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:16.339-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:16.339-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:16.339-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:59:16.897-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:16.924-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:16.978-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:16.978-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:16.978-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:59:17.535-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:17.593-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:17.622-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:17.622-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:17.622-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:59:18.179-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:18.206-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:18.264-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:18.264-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:18.264-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:59:18.820-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:18.849-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:18.902-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:18.902-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:18.902-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:59:19.430-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:19.485-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:19.543-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:19.543-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:19.543-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:59:20.097-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:20.124-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:20.179-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:20.180-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:20.180-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:59:20.733-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:20.791-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:20.820-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:20.820-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:20.820-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:59:20.820-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:59:22.849-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:22.904-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:22.958-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:22.958-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:22.958-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:59:23.514-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:23.543-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:23.600-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:23.600-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:23.600-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:59:24.156-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:24.213-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:24.243-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:24.243-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:24.243-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:59:24.798-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:24.826-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:24.883-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:24.883-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:24.883-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:59:25.440-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:25.498-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:25.525-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:25.525-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:25.525-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:59:26.054-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:26.083-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:26.109-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:26.109-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:59:26.109-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:59:26.637-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:26.692-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:26.746-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:26.746-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:26.746-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:59:27.274-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:27.329-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:27.385-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:27.385-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:27.385-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:59:27.912-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:27.968-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:28.023-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:28.024-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:28.024-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:59:28.553-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:28.608-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:28.656-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:28.656-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T13:59:28.656-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:59:29.185-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:29.212-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:29.241-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:29.241-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:29.241-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:59:29.241-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:29.241-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:31.241-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:33.241-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:35.241-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:36.110-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T13:59:37.241-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T13:59:37.241-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:39.295-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:39.324-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:39.382-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:39.382-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:39.382-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T13:59:39.938-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:39.993-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:40.020-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:40.020-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:40.020-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T13:59:40.574-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:40.628-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:40.655-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:40.655-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:40.655-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T13:59:41.213-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:41.268-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:41.295-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:41.295-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:41.295-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T13:59:41.851-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:41.905-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:41.932-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:41.932-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:41.932-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T13:59:42.490-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:42.519-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:42.576-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:42.576-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:42.577-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T13:59:43.130-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:43.184-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:43.211-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:43.211-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:43.211-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T13:59:43.765-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:43.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:43.846-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:43.846-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:43.846-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T13:59:44.402-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:44.431-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:44.485-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:44.485-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:44.485-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T13:59:45.039-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:45.094-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:45.124-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:45.124-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:45.124-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T13:59:45.677-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:45.706-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:45.764-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:45.764-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:45.764-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T13:59:46.167-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:46.196-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:46.250-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:46.250-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:59:46.250-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T13:59:46.321-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:46.350-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:46.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:46.406-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:46.406-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T13:59:46.960-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:46.989-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:47.044-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:47.045-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:47.045-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T13:59:47.598-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:47.654-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:47.681-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:47.681-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:47.681-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T13:59:48.239-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:48.268-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:48.323-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:48.324-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:48.324-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T13:59:48.877-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:48.933-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:48.960-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:48.960-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:48.960-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T13:59:49.516-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:49.545-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:49.601-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:49.601-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:49.601-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T13:59:50.158-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:50.185-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:50.241-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:50.241-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:50.241-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T13:59:50.241-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T13:59:52.295-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:52.324-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:52.381-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:52.381-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:52.381-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T13:59:52.911-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:52.968-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:53.022-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:53.022-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:53.022-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T13:59:53.575-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:53.629-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:53.658-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:53.658-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:53.658-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T13:59:54.214-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:54.268-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:54.297-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:54.297-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:54.297-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T13:59:54.824-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:54.882-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:54.937-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:54.937-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:54.937-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T13:59:55.495-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:55.550-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:55.578-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:55.578-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:55.578-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T13:59:56.135-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:56.191-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:56.218-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:56.218-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:56.218-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T13:59:56.279-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:56.335-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:56.390-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:56.390-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T13:59:56.390-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T13:59:56.774-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:56.803-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:56.857-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:56.857-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:56.857-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T13:59:57.411-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:57.438-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:57.493-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:57.493-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:57.493-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T13:59:58.021-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T13:59:58.076-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T13:59:58.130-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T13:59:58.130-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T13:59:58.130-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T13:59:58.130-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T13:59:59.241-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:00.130-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:02.130-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:04.131-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:06.131-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:06.390-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:00:08.131-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:00:08.131-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:10.189-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:10.216-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:10.272-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:10.272-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:10.272-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:00:10.827-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:10.855-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:10.910-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:10.910-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:10.910-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:00:11.468-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:11.523-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:11.552-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:11.552-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:11.553-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:00:12.080-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:12.135-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:12.189-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:12.189-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:12.189-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:00:12.718-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:12.774-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:12.827-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:12.827-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:12.827-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:00:13.355-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:13.408-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:13.464-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:13.464-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:13.464-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:00:13.993-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:14.049-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:14.106-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:14.106-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:14.106-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:00:14.633-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:14.689-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:14.744-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:14.744-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:14.744-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:00:15.300-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:15.356-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:15.385-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:15.385-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:15.385-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:00:15.942-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:15.970-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:16.024-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:16.024-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:16.024-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:00:16.448-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:16.477-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:16.534-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:16.534-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:00:16.535-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:00:17.090-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:17.119-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:17.175-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:17.175-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:17.175-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:00:17.704-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:17.762-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:17.817-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:17.817-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:17.817-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:00:18.371-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:18.398-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:18.454-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:18.454-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:18.454-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:00:18.981-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:19.039-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:19.094-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:19.094-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:19.094-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:00:19.650-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:19.707-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:19.736-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:19.736-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:19.736-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:00:20.266-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:20.321-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:20.377-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:20.377-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:20.377-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:00:20.932-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:20.961-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:21.017-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:21.017-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:21.017-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:00:21.017-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:00:23.075-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:23.130-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:23.159-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:23.159-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:23.159-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:00:23.715-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:23.744-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:23.798-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:23.798-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:23.798-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:00:24.325-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:24.379-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:24.433-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:24.433-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:24.433-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:00:24.990-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:25.018-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:25.075-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:25.075-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:25.075-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:00:25.602-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:25.660-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:25.717-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:25.717-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:25.717-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:00:26.246-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:26.300-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:26.355-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:26.355-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:26.355-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:00:26.590-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:26.617-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:26.675-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:26.675-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:00:26.675-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:00:26.909-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:26.967-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:26.996-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:26.996-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:26.996-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:00:27.551-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:27.605-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:27.634-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:27.634-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:27.634-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:00:28.163-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:28.220-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:28.276-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:28.276-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:28.276-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:00:28.831-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:28.885-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:28.913-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:28.913-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:28.913-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:00:29.271-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:29.326-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:29.382-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:29.382-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:00:29.382-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:00:29.382-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:29.413-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:31.413-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:33.413-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:35.413-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:36.675-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:00:37.414-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:00:37.414-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:39.471-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:39.500-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:39.558-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:39.558-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:39.558-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:00:40.114-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:40.169-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:40.196-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:40.196-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:40.196-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:00:40.750-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:40.804-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:40.833-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:40.833-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:40.833-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:00:41.390-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:41.418-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:41.475-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:41.475-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:41.475-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:00:42.004-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:42.060-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:42.114-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:42.114-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:42.114-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:00:42.641-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:42.695-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:42.748-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:42.749-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:42.749-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:00:43.304-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:43.358-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:43.387-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:43.387-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:43.387-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:00:43.942-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:43.969-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:44.027-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:44.027-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:44.027-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:00:44.581-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:44.636-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:44.663-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:44.663-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:44.663-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:00:45.221-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:45.248-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:45.305-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:45.306-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:45.306-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:00:45.833-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:45.887-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:45.944-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:45.944-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:45.944-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:00:46.500-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:46.527-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:46.583-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:46.583-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:46.583-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:00:46.731-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:46.758-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:46.814-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:46.814-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:00:46.814-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:00:47.136-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:47.192-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:47.219-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:47.219-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:47.219-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:00:47.746-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:47.802-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:47.858-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:47.858-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:47.858-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:00:48.413-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:48.442-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:48.498-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:48.498-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:48.499-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:00:49.028-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:49.085-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:49.141-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:49.141-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:49.141-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:00:49.698-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:49.727-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:49.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:49.783-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:49.783-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:00:50.339-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:50.394-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:50.423-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:50.423-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:50.423-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:00:50.423-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:00:52.479-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:52.506-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:52.562-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:52.562-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:52.562-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:00:53.119-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:53.177-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:53.206-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:53.206-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:53.206-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:00:53.735-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:53.791-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:53.848-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:53.848-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:53.848-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:00:54.402-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:54.459-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:54.487-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:54.487-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:54.487-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:00:55.014-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:55.070-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:55.127-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:55.127-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:55.127-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:00:55.685-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:55.740-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:55.767-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:55.767-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:55.767-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:00:56.323-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:56.352-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:56.408-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:56.408-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:56.408-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:00:56.843-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:56.900-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:56.956-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:56.956-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:00:56.956-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:00:57.510-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:57.539-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:57.594-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:57.594-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:57.594-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:00:58.148-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:58.204-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:58.231-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:58.231-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:58.231-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:00:58.760-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:00:58.814-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:00:58.870-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:00:58.870-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:00:58.870-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:00:58.870-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:00:59.382-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:00.870-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:02.870-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:04.870-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:06.871-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:06.956-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:01:08.871-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:01:08.871-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:10.900-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:10.954-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:11.008-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:11.008-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:11.008-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:01:11.537-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:11.594-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:11.650-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:11.650-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:11.650-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:01:12.205-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:12.233-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:12.286-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:12.286-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:12.286-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:01:12.844-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:12.871-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:12.927-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:12.927-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:12.927-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:01:13.456-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:13.512-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:13.568-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:13.568-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:13.568-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:01:14.123-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:14.179-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:14.208-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:14.208-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:14.208-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:01:14.762-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:14.791-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:14.847-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:14.847-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:14.847-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:01:15.403-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:15.456-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:15.483-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:15.484-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:15.484-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:01:16.037-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:16.095-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:16.124-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:16.124-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:16.124-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:01:16.652-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:16.709-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:16.767-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:16.767-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:16.767-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:01:16.983-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:17.039-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:17.094-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:17.094-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:01:17.094-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:01:17.296-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:17.353-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:17.410-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:17.410-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:17.410-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:01:17.938-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:17.993-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:18.049-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:18.049-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:18.049-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:01:18.605-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:18.634-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:18.689-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:18.689-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:18.689-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:01:19.247-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:19.276-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:19.330-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:19.330-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:19.330-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:01:19.857-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:19.913-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:19.968-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:19.968-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:19.968-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:01:20.497-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:20.551-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:20.605-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:20.605-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:20.605-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:01:21.134-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:21.188-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:21.245-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:21.245-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:21.245-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:01:21.774-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:21.830-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:21.885-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:21.885-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:21.885-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:01:21.885-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:01:23.939-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:23.967-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:24.022-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:24.022-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:24.022-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:01:24.578-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:24.605-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:24.663-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:24.663-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:24.663-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:01:25.218-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:25.276-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:25.303-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:25.303-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:25.303-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:01:25.861-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:25.916-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:25.945-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:25.945-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:25.945-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:01:26.474-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:26.528-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:26.582-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:26.582-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:26.582-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:01:27.109-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:27.148-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:27.165-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:27.165-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:27.165-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:01:27.692-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:27.748-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:27.803-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:27.803-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:27.803-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:01:28.361-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:28.416-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:28.443-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:28.443-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:28.443-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:01:28.972-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:29.028-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:29.085-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:29.085-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:29.085-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:01:29.411-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:29.465-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:29.520-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:29.521-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:01:29.521-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:01:29.641-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:29.668-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:29.726-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:29.726-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:29.726-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:01:29.726-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:30.021-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:31.726-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:33.726-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:35.726-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:37.165-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:01:37.727-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:01:37.727-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:01:39.754-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:39.810-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:39.867-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:39.867-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:39.867-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:01:40.425-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:40.452-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:40.506-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:40.506-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:40.506-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:01:41.035-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:41.090-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:41.147-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:41.147-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:41.147-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:01:41.674-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:41.728-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:41.785-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:41.786-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:41.786-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:01:42.340-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:42.395-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:42.424-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:42.424-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:42.424-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:01:42.980-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:43.009-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:43.066-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:43.066-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:43.066-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:01:43.622-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:43.676-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:43.703-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:43.703-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:43.703-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:01:44.230-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:44.286-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:44.341-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:44.341-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:44.341-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:01:44.895-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:44.949-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:44.977-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:44.977-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:44.977-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:01:45.535-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:45.589-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:45.618-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:45.618-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:45.618-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:01:46.175-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:46.204-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:46.262-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:46.262-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:46.262-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:01:46.789-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:46.846-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:46.902-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:46.902-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:46.902-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:01:47.221-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:47.278-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:47.307-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:47.308-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:01:47.308-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:01:47.460-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:47.489-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:47.546-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:47.546-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:47.546-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:01:48.102-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:48.157-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:48.186-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:48.186-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:48.186-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:01:48.714-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:48.769-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:48.825-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:48.825-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:48.825-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:01:49.381-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:49.410-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:49.465-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:49.465-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:49.465-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:01:50.022-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:50.080-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:50.109-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:50.109-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:50.109-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:01:50.667-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:50.722-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:50.749-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:50.749-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:50.750-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:01:50.750-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:01:52.805-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:52.834-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:52.888-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:52.888-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:52.888-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:01:53.416-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:53.471-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:53.527-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:53.527-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:53.527-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:01:54.082-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:54.111-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:54.169-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:54.169-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:54.169-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:01:54.696-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:54.752-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:54.809-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:54.809-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:54.809-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:01:55.337-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:55.392-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:55.446-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:55.446-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:55.446-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:01:55.973-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:56.029-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:56.082-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:56.083-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:56.083-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:01:56.640-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:56.669-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:56.723-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:56.723-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:56.723-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:01:57.277-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:57.334-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:57.337-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:57.337-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:01:57.337-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:01:57.892-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:57.948-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:57.975-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:57.975-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:57.975-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:01:58.502-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:58.558-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:58.614-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:58.614-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:58.614-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:01:59.141-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:01:59.196-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:01:59.252-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:01:59.252-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:01:59.252-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:01:59.252-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:00.021-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:01.252-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:03.252-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:05.253-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:07.253-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:07.337-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:02:09.253-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:02:09.253-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:11.307-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:11.365-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:11.394-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:11.394-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:11.394-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:02:11.921-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:11.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:12.031-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:12.031-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:12.031-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:02:12.560-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:12.615-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:12.669-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:12.669-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:12.669-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:02:13.225-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:13.280-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:13.309-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:13.309-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:13.310-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:02:13.837-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:13.892-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:13.948-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:13.948-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:13.948-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:02:14.504-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:14.559-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:14.588-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:14.588-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:14.588-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:02:15.142-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:15.198-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:15.227-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:15.227-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:15.227-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:02:15.756-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:15.812-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:15.869-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:15.869-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:15.869-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:02:16.398-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:16.452-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:16.508-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:16.508-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:16.508-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:02:17.066-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:17.121-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:17.150-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:17.150-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:17.150-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:02:17.393-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:17.449-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:17.476-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:17.476-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:02:17.476-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:02:17.708-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:17.737-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:17.794-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:17.794-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:17.794-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:02:18.321-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:18.375-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:18.429-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:18.429-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:18.429-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:02:18.958-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:19.014-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:19.071-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:19.071-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:19.071-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:02:19.600-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:19.654-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:19.710-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:19.710-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:19.710-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:02:20.265-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:20.293-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:20.348-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:20.348-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:20.348-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:02:20.905-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:20.934-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:20.988-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:20.988-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:20.988-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:02:21.517-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:21.575-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:21.632-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:21.632-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:21.632-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:02:22.188-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:22.215-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:22.270-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:22.270-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:22.270-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:02:22.271-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:02:24.328-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:24.384-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:24.411-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:24.411-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:24.411-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:02:24.940-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:24.998-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:25.052-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:25.052-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:25.052-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:02:25.581-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:25.638-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:25.695-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:25.696-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:25.696-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:02:26.251-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:26.307-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:26.336-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:26.336-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:26.336-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:02:26.865-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:26.922-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:26.976-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:26.976-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:26.976-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:02:27.503-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:27.532-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:27.557-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:27.557-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:27.557-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:02:28.084-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:28.138-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:28.194-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:28.194-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:28.194-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:02:28.749-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:28.778-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:28.832-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:28.832-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:28.832-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:02:29.361-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:29.419-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:29.474-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:29.474-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:29.474-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:02:30.028-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:30.050-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:30.084-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:30.084-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:30.084-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:02:30.637-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:30.639-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:30.665-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:30.665-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:02:30.665-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:02:30.665-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:30.665-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:32.665-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:34.665-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:36.665-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:37.557-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:02:38.666-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:02:38.666-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:02:40.723-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:40.751-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:40.804-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:40.804-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:40.804-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:02:41.332-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:41.387-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:41.443-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:41.443-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:41.443-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:02:41.972-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:42.030-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:42.087-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:42.087-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:42.087-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:02:42.614-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:42.672-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:42.726-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:42.726-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:42.726-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:02:43.283-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:43.337-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:43.364-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:43.364-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:43.364-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:02:43.918-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:43.947-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:44.003-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:44.003-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:44.003-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:02:44.530-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:44.588-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:44.643-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:44.643-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:44.643-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:02:45.197-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:45.226-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:45.280-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:45.280-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:45.280-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:02:45.838-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:45.865-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:45.920-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:45.920-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:45.921-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:02:46.448-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:46.502-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:46.557-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:46.557-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:46.557-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:02:47.113-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:47.169-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:47.196-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:47.196-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:47.196-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:02:47.586-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:47.642-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:47.698-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:47.698-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:02:47.698-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:02:48.227-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:48.281-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:48.338-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:48.338-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:48.338-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:02:48.894-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:48.921-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:48.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:48.977-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:48.977-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:02:49.506-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:49.561-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:49.617-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:49.617-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:49.617-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:02:50.146-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:50.200-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:50.256-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:50.256-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:50.256-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:02:50.811-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:50.839-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:50.892-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:50.892-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:50.892-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:02:51.450-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:51.507-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:51.536-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:51.537-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:51.537-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:02:51.537-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:02:53.566-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:53.620-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:53.677-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:53.677-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:53.677-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:02:54.233-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:54.288-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:54.315-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:54.316-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:54.316-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:02:54.871-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:54.929-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:54.956-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:54.956-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:54.956-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:02:55.513-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:55.569-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:55.598-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:55.598-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:55.598-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:02:56.154-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:56.183-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:56.240-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:56.240-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:56.240-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:02:56.798-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:56.854-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:56.883-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:56.883-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:56.883-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:02:57.440-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:57.467-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:57.525-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:57.525-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:57.525-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:02:57.725-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:57.778-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:57.834-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:57.834-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:02:57.834-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:02:58.052-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:58.108-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:58.163-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:58.163-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:58.163-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:02:58.691-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:58.746-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:58.800-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:58.800-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:58.800-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:02:59.358-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:02:59.387-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:02:59.444-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:02:59.444-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:02:59.444-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:02:59.998-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:00.027-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:00.081-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:00.081-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:00.081-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:03:00.081-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:00.665-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:02.081-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:04.081-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:06.082-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:07.834-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:03:08.082-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:03:08.082-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:10.138-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:10.193-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:10.222-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:10.222-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:10.222-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:03:10.780-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:10.834-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:10.863-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:10.863-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:10.863-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:03:11.392-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:11.449-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:11.507-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:11.507-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:11.507-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:03:12.061-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:12.090-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:12.146-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:12.146-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:12.146-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:03:12.703-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:12.730-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:12.786-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:12.786-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:12.786-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:03:13.340-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:13.367-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:13.423-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:13.423-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:13.423-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:03:13.952-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:14.010-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:14.067-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:14.067-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:14.067-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:03:14.596-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:14.650-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:14.708-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:14.708-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:14.708-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:03:15.235-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:15.291-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:15.346-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:15.346-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:15.346-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:03:15.904-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:15.962-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:15.991-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:15.991-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:15.991-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:03:16.518-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:16.574-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:16.629-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:16.629-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:16.629-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:03:17.185-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:17.239-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:17.268-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:17.268-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:17.268-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:03:17.824-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:17.863-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:17.881-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:17.881-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:17.881-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:03:18.437-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:18.466-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:18.524-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:18.524-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:18.524-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:03:19.081-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:19.139-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:19.168-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:19.168-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:19.168-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:03:19.722-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:19.749-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:19.803-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:19.803-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:19.803-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:03:20.360-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:20.388-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:20.445-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:20.445-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:20.445-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:03:20.973-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:21.028-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:21.086-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:21.086-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:21.086-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:03:21.086-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:03:23.141-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:23.197-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:23.224-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:23.224-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:23.224-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:03:23.780-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:23.834-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:23.863-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:23.863-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:23.863-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:03:24.419-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:24.448-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:24.505-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:24.506-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:24.506-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:03:25.061-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:25.090-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:25.148-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:25.148-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:25.148-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:03:25.675-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:25.729-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:25.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:25.783-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:25.783-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:03:26.337-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:26.393-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:26.422-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:26.422-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:26.422-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:03:26.979-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:27.035-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:27.064-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:27.064-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:27.064-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:03:27.593-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:27.649-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:27.706-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:27.707-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:27.707-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:03:27.910-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:27.966-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:28.020-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:28.020-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:03:28.020-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:03:28.262-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:28.319-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:28.347-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:28.347-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:28.347-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:03:28.876-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:28.932-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:28.987-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:28.987-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:28.987-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:03:29.517-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:29.570-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:29.628-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:29.628-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:29.628-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:03:29.628-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:30.665-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:31.628-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:33.628-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:35.629-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:37.629-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:38.020-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:03:39.629-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:03:39.629-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:03:41.683-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:41.710-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:41.768-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:41.768-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:41.768-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:03:42.326-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:42.381-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:42.409-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:42.409-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:42.409-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:03:42.936-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:42.993-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:43.051-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:43.051-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:43.051-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:03:43.605-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:43.660-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:43.688-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:43.688-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:43.688-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:03:44.245-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:44.274-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:44.330-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:44.330-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:44.330-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:03:44.884-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:44.913-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:44.969-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:44.969-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:44.969-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:03:45.523-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:45.576-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:45.604-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:45.604-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:45.604-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:03:46.131-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:46.187-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:46.242-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:46.242-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:46.242-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:03:46.770-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:46.824-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:46.879-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:46.879-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:46.879-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:03:47.408-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:47.464-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:47.520-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:47.520-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:47.520-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:03:48.075-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:48.076-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:48.103-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:48.103-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:48.103-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:03:48.657-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:48.712-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:48.740-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:48.740-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:48.740-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:03:49.295-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:49.349-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:49.378-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:49.378-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:49.378-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:03:49.932-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:49.961-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:50.019-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:50.019-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:50.019-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:03:50.576-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:50.632-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:50.659-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:50.659-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:50.659-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:03:51.213-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:51.242-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:51.298-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:51.298-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:51.298-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:03:51.852-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:51.909-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:51.938-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:51.938-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:51.938-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:03:52.496-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:52.523-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:52.579-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:52.579-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:52.579-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:03:52.579-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:03:54.633-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:54.660-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:54.716-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:54.716-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:54.716-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:03:55.271-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:55.327-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:55.354-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:55.354-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:55.354-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:03:55.910-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:55.939-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:55.996-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:55.996-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:55.996-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:03:56.550-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:56.577-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:56.633-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:56.633-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:56.633-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:03:57.189-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:57.244-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:57.272-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:57.272-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:57.272-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:03:57.827-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:57.881-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:57.910-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:57.910-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:57.910-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:03:58.160-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:58.218-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:58.247-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:58.247-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:03:58.247-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:03:58.466-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:58.521-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:58.550-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:58.550-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:58.550-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:03:59.080-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:59.135-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:59.191-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:59.191-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:59.191-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:03:59.718-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:03:59.774-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:03:59.828-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:03:59.828-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:03:59.828-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:04:00.355-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:00.413-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:00.468-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:00.468-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:00.468-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:04:00.721-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:00.778-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:00.806-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:00.806-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:04:00.806-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:04:00.806-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:00.969-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:02.969-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:04.969-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:06.969-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:08.247-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:04:08.970-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:04:08.970-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:11.023-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:11.051-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:11.106-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:11.106-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:11.106-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:04:11.662-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:11.716-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:11.745-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:11.745-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:11.745-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:04:12.272-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:12.328-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:12.382-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:12.382-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:12.382-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:04:12.939-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:12.968-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:13.026-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:13.026-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:13.026-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:04:13.582-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:13.611-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:13.668-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:13.668-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:13.668-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:04:14.224-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:14.279-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:14.307-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:14.307-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:14.307-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:04:14.862-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:14.916-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:14.943-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:14.943-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:14.943-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:04:15.499-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:15.555-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:15.584-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:15.584-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:15.584-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:04:16.139-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:16.195-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:16.222-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:16.222-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:16.222-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:04:16.780-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:16.807-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:16.863-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:16.863-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:16.863-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:04:17.418-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:17.474-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:17.501-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:17.501-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:17.501-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:04:18.059-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:18.088-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:18.141-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:18.141-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:18.141-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:04:18.304-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:18.358-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:18.387-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:18.387-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:04:18.387-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:04:18.697-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:18.753-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:18.782-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:18.782-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:18.782-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:04:19.338-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:19.393-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:19.422-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:19.422-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:19.422-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:04:19.978-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:20.007-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:20.063-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:20.063-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:20.063-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:04:20.619-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:20.648-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:20.705-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:20.705-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:20.705-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:04:21.261-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:21.288-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:21.346-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:21.346-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:21.346-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:04:21.901-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:21.955-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:21.984-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:21.984-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:21.984-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:04:21.984-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:04:24.038-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:24.092-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:24.119-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:24.119-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:24.119-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:04:24.677-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:24.706-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:24.760-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:24.760-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:24.760-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:04:25.314-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:25.369-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:25.396-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:25.397-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:25.397-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:04:25.926-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:25.981-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:26.039-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:26.039-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:26.039-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:04:26.568-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:26.626-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:26.679-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:26.679-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:26.679-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:04:27.235-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:27.289-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:27.318-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:27.318-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:27.318-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:04:27.876-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:27.931-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:27.959-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:27.959-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:27.959-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:04:28.443-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:28.486-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:28.500-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:28.500-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:04:28.500-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:04:29.054-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:29.110-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:29.139-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:29.139-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:29.139-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:04:29.668-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:29.724-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:29.779-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:29.779-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:29.779-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:04:30.306-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:30.362-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:30.418-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:30.418-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:30.418-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:04:30.418-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:30.806-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:32.418-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:34.418-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:36.418-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:38.419-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:38.501-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:04:40.419-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:04:40.419-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:04:42.475-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:42.532-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:42.559-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:42.560-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:42.560-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:04:43.117-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:43.173-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:43.200-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:43.200-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:43.200-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:04:43.729-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:43.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:43.836-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:43.836-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:43.836-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:04:44.392-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:44.450-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:44.477-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:44.477-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:44.477-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:04:45.032-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:45.086-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:45.115-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:45.115-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:45.115-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:04:45.671-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:45.700-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:45.754-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:45.754-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:45.754-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:04:46.310-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:46.365-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:46.394-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:46.394-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:46.394-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:04:46.950-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:46.979-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:47.035-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:47.035-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:47.035-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:04:47.590-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:47.617-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:47.671-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:47.671-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:47.671-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:04:48.225-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:48.254-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:48.312-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:48.312-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:48.312-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:04:48.554-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:48.610-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:48.639-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:48.639-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:04:48.639-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:04:48.866-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:48.895-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:48.950-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:48.950-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:48.950-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:04:49.504-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:49.560-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:49.589-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:49.589-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:49.589-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:04:50.144-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:50.198-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:50.227-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:50.227-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:50.227-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:04:50.754-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:50.808-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:50.865-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:50.865-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:50.865-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:04:51.395-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:51.452-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:51.507-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:51.507-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:51.507-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:04:52.063-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:52.090-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:52.144-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:52.144-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:52.144-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:04:52.700-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:52.756-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:52.785-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:52.785-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:52.785-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:04:53.339-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:53.368-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:53.423-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:53.423-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:53.423-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:04:53.423-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:04:55.479-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:55.506-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:55.560-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:55.560-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:55.560-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:04:56.117-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:56.147-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:56.200-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:56.200-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:56.200-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:04:56.728-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:56.783-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:56.841-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:56.841-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:56.841-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:04:57.395-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:57.422-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:57.477-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:57.478-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:57.478-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:04:58.033-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:58.089-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:58.118-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:58.118-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:58.118-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:04:58.668-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:58.674-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:58.724-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:58.724-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:04:58.724-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:04:59.278-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:59.334-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:04:59.363-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:04:59.363-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:04:59.363-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:04:59.918-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:04:59.976-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:00.003-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:00.003-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:00.003-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:05:00.560-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:00.618-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:00.645-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:00.645-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:00.645-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:05:00.862-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:00.917-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:00.944-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:00.944-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:05:00.944-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:05:01.172-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:01.228-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:01.285-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:01.285-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:01.285-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:05:01.286-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:01.445-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:03.286-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:05.286-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:07.286-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:08.724-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:05:09.286-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:05:09.286-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:11.344-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:11.398-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:11.427-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:11.427-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:11.427-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:05:11.981-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:12.010-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:12.065-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:12.065-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:12.065-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:05:12.595-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:12.652-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:12.707-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:12.708-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:12.708-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:05:13.237-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:13.292-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:13.350-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:13.350-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:13.350-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:05:13.905-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:13.934-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:13.988-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:13.988-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:13.988-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:05:14.544-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:14.600-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:14.627-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:14.627-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:14.627-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:05:15.181-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:15.234-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:15.263-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:15.263-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:15.263-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:05:15.819-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:15.876-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:15.904-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:15.904-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:15.904-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:05:16.459-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:16.488-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:16.544-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:16.544-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:16.544-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:05:17.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:17.155-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:17.185-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:17.185-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:17.185-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:05:17.738-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:17.767-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:17.821-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:17.821-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:17.821-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:05:18.379-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:18.408-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:18.462-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:18.462-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:18.462-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:05:18.780-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:18.809-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:18.862-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:18.862-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:05:18.862-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:05:19.017-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:19.075-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:19.102-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:19.102-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:19.102-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:05:19.658-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:19.713-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:19.740-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:19.741-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:19.741-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:05:20.298-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:20.354-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:20.381-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:20.381-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:20.381-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:05:20.937-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:20.994-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:21.021-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:21.021-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:21.021-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:05:21.577-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:21.604-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:21.660-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:21.660-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:21.660-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:05:22.187-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:22.245-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:22.302-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:22.302-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:22.302-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:05:22.302-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:05:24.330-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:24.384-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:24.441-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:24.441-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:24.441-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:05:24.968-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:25.024-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:25.081-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:25.081-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:25.081-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:05:25.637-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:25.693-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:25.722-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:25.722-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:25.722-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:05:26.249-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:26.305-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:26.360-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:26.360-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:26.360-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:05:26.916-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:26.973-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:27.002-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:27.002-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:27.002-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:05:27.560-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:27.587-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:27.645-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:27.645-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:27.645-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:05:28.200-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:28.254-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:28.283-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:28.283-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:28.283-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:05:28.812-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:28.866-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:28.920-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:28.920-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:05:28.920-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:05:29.447-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:29.505-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:29.560-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:29.560-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:29.560-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:05:30.114-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:30.170-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:30.199-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:30.199-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:30.199-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:05:30.754-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:30.812-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:30.841-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:30.841-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:30.841-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:05:30.841-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:31.445-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:32.841-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:34.841-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:36.842-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:38.842-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:38.920-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:05:40.842-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:05:40.842-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:05:42.869-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:42.925-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:42.980-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:42.980-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:42.980-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:05:43.536-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:43.563-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:43.619-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:43.619-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:43.619-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:05:44.173-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:44.230-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:44.257-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:44.258-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:44.258-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:05:44.815-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:44.842-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:44.898-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:44.898-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:44.898-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:05:45.452-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:45.479-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:45.534-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:45.534-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:45.534-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:05:46.092-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:46.145-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:46.173-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:46.173-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:46.173-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:05:46.700-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:46.757-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:46.813-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:46.813-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:46.813-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:05:47.368-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:47.426-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:47.455-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:47.455-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:47.455-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:05:47.984-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:48.041-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:48.097-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:48.097-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:48.097-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:05:48.653-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:48.680-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:48.735-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:48.735-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:48.735-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:05:48.976-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:49.031-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:49.060-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:49.060-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:05:49.060-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:05:49.265-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:49.320-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:49.376-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:49.376-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:49.376-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:05:49.905-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:49.960-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:50.016-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:50.016-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:50.016-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:05:50.543-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:50.599-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:50.652-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:50.652-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:50.652-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:05:51.180-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:51.237-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:51.295-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:51.295-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:51.295-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:05:51.850-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:51.906-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:51.935-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:51.935-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:51.935-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:05:52.491-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:52.520-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:52.577-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:52.577-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:52.577-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:05:53.135-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:53.190-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:53.217-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:53.217-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:53.217-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:05:53.773-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:53.800-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:53.854-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:53.854-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:53.854-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:05:53.854-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:05:55.908-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:55.964-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:55.993-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:55.993-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:55.993-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:05:56.550-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:56.607-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:56.635-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:56.635-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:56.635-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:05:57.189-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:57.244-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:57.273-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:57.273-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:57.273-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:05:57.801-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:57.858-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:57.914-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:57.914-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:57.914-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:05:58.471-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:58.529-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:58.558-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:58.558-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:58.558-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:05:59.112-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:59.116-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:59.141-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:59.141-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:59.141-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:05:59.695-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:05:59.722-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:05:59.778-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:05:59.778-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:05:59.778-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:06:00.307-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:00.364-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:00.420-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:00.420-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:00.420-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:06:00.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:01.033-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:01.060-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:01.060-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:01.060-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:06:01.472-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:01.526-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:01.579-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:01.580-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:06:01.580-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:06:02.107-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:02.135-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:02.162-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:02.163-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:06:02.163-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:06:02.163-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:02.163-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:04.163-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:06.163-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:08.163-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:09.141-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:06:10.164-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:06:10.164-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:12.217-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:12.246-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:12.302-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:12.302-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:12.302-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:06:12.858-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:12.913-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:12.940-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:12.940-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:12.940-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:06:13.496-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:13.552-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:13.581-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:13.581-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:13.581-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:06:14.136-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:14.190-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:14.217-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:14.217-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:14.217-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:06:14.775-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:14.832-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:14.859-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:14.859-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:14.859-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:06:15.387-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:15.442-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:15.498-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:15.498-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:15.498-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:06:16.052-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:16.109-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:16.138-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:16.138-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:16.138-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:06:16.694-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:16.749-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:16.778-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:16.778-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:16.778-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:06:17.334-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:17.389-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:17.416-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:17.416-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:17.416-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:06:17.972-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:17.999-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:18.053-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:18.053-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:18.053-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:06:18.580-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:18.634-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:18.690-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:18.690-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:18.690-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:06:19.197-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:19.224-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:19.245-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:19.245-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:19.245-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:06:19.799-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:19.828-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:19.882-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:19.882-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:19.882-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:06:20.409-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:20.465-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:20.522-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:20.522-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:20.522-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:06:21.078-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:21.132-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:21.159-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:21.159-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:21.159-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:06:21.688-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:21.746-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:21.799-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:21.799-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:21.799-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:06:22.355-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:22.382-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:22.438-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:22.438-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:22.438-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:06:22.995-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:23.053-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:23.082-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:23.082-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:23.082-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:06:23.082-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:06:25.136-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:25.165-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:25.220-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:25.220-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:25.221-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:06:25.776-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:25.832-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:25.861-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:25.861-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:25.861-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:06:26.388-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:26.444-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:26.497-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:26.497-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:26.498-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:06:27.025-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:27.078-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:27.134-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:27.134-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:27.134-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:06:27.690-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:27.717-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:27.772-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:27.772-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:27.772-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:06:28.300-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:28.355-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:28.409-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:28.409-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:28.409-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:06:28.965-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:29.022-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:29.049-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:29.049-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:29.049-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:06:29.301-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:29.355-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:29.382-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:29.382-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:06:29.382-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:06:29.603-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:29.632-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:29.686-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:29.686-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:29.686-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:06:30.215-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:30.269-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:30.326-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:30.326-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:30.326-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:06:30.882-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:30.936-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:30.965-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:30.965-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:30.965-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:06:31.522-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:31.551-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:31.609-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:31.609-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:31.609-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:06:31.609-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:32.163-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:33.609-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:35.609-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:37.609-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:39.382-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:06:39.610-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:06:39.610-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:06:41.639-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:41.693-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:41.746-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:41.746-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:41.746-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:06:42.300-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:42.356-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:42.385-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:42.385-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:42.385-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:06:42.940-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:42.996-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:43.025-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:43.025-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:43.025-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:06:43.552-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:43.608-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:43.664-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:43.664-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:43.664-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:06:44.193-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:44.250-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:44.304-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:44.304-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:44.304-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:06:44.860-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:44.917-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:44.946-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:44.946-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:44.946-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:06:45.475-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:45.531-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:45.587-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:45.587-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:45.587-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:06:46.114-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:46.171-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:46.227-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:46.227-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:46.227-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:06:46.782-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:46.809-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:46.863-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:46.863-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:46.863-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:06:47.392-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:47.450-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:47.507-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:47.507-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:47.507-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:06:48.061-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:48.088-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:48.144-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:48.144-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:48.144-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:06:48.700-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:48.753-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:48.780-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:48.781-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:48.781-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:06:49.334-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:49.390-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:49.409-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:49.409-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:06:49.409-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:06:49.939-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:49.994-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:50.050-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:50.050-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:50.050-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:06:50.579-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:50.633-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:50.686-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:50.686-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:50.686-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:06:51.215-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:51.269-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:51.327-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:51.327-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:51.327-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:06:51.882-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:51.938-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:51.967-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:51.967-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:51.967-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:06:52.496-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:52.552-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:52.607-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:52.607-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:52.607-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:06:52.607-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:06:54.663-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:54.690-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:54.746-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:54.746-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:54.746-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:06:55.301-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:55.359-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:55.388-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:55.388-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:55.388-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:06:55.915-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:55.971-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:56.024-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:56.024-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:56.024-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:06:56.552-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:56.607-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:56.665-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:56.665-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:56.665-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:06:57.220-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:57.276-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:57.305-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:57.305-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:57.305-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:06:57.860-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:57.888-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:57.945-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:57.945-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:57.945-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:06:58.472-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:58.528-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:58.583-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:58.583-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:58.583-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:06:59.137-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:59.164-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:59.220-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:59.220-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:59.220-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:06:59.436-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:59.492-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:59.549-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:59.549-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:06:59.549-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:06:59.776-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:06:59.803-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:06:59.858-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:06:59.858-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:06:59.858-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:07:00.385-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:00.441-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:00.497-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:00.497-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:00.497-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:07:01.054-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:01.108-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:01.135-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:01.135-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:01.135-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:07:01.135-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:02.163-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:03.136-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:05.136-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:07.136-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:09.136-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:09.549-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:07:11.136-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:07:11.136-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:13.192-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:13.221-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:13.277-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:13.277-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:13.277-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:07:13.834-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:13.861-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:13.917-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:13.917-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:13.917-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:07:14.471-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:14.524-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:14.552-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:14.552-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:14.552-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:07:15.107-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:15.165-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:15.194-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:15.194-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:15.194-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:07:15.723-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:15.778-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:15.832-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:15.832-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:15.832-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:07:16.388-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:16.444-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:16.471-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:16.471-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:16.471-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:07:17.025-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:17.080-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:17.109-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:17.109-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:17.109-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:07:17.665-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:17.721-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:17.748-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:17.748-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:17.748-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:07:18.305-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:18.359-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:18.386-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:18.386-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:18.386-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:07:18.913-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:18.967-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:19.023-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:19.023-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:19.023-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:07:19.576-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:19.577-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:19.632-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:19.632-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:19.632-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:07:20.186-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:20.240-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:20.267-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:20.267-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:20.267-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:07:20.821-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:20.876-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:20.905-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:20.905-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:20.905-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:07:21.461-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:21.517-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:21.544-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:21.544-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:21.544-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:07:22.073-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:22.129-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:22.182-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:22.183-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:22.183-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:07:22.712-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:22.767-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:22.823-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:22.823-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:22.823-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:07:23.378-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:23.406-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:23.461-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:23.461-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:23.461-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:07:23.990-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:24.044-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:24.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:24.100-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:24.100-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:07:24.100-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:07:26.154-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:26.209-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:26.236-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:26.236-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:26.236-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:07:26.790-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:26.847-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:26.876-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:26.876-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:26.876-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:07:27.432-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:27.459-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:27.513-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:27.513-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:27.513-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:07:28.067-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:28.094-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:28.149-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:28.149-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:28.149-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:07:28.705-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:28.760-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:28.788-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:28.788-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:28.788-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:07:29.315-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:29.369-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:29.424-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:29.424-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:29.424-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:07:29.690-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:29.717-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:29.774-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:29.774-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:07:29.774-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:07:29.980-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:30.035-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:30.065-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:30.065-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:30.065-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:07:30.622-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:30.676-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:30.705-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:30.705-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:30.705-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:07:31.262-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:31.290-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:31.345-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:31.345-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:31.345-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:07:31.899-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:31.928-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:31.984-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:31.984-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:31.984-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:07:32.220-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:32.276-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:32.303-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:32.303-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:07:32.303-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:07:32.303-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:32.484-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:34.484-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:36.484-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:38.484-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:39.774-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:07:40.485-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:07:40.485-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:07:42.539-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:42.568-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:42.621-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:42.621-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:42.621-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:07:43.177-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:43.206-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:43.264-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:43.264-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:43.264-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:07:43.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:43.850-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:43.908-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:43.908-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:43.908-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:07:44.437-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:44.492-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:44.550-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:44.550-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:44.550-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:07:45.103-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:45.133-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:45.186-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:45.186-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:45.186-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:07:45.744-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:45.800-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:45.827-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:45.827-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:45.827-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:07:46.381-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:46.436-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:46.465-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:46.465-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:46.465-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:07:46.994-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:47.050-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:47.104-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:47.104-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:47.104-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:07:47.659-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:47.715-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:47.742-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:47.742-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:47.742-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:07:48.298-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:48.353-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:48.382-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:48.382-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:48.382-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:07:48.938-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:48.994-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:49.021-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:49.021-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:49.021-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:07:49.578-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:49.634-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:49.661-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:49.661-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:49.661-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:07:49.832-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:49.889-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:49.916-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:49.916-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:07:49.916-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:07:50.217-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:50.246-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:50.301-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:50.302-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:50.302-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:07:50.831-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:50.886-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:50.942-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:50.942-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:50.942-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:07:51.497-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:51.551-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:51.578-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:51.578-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:51.578-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:07:52.136-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:52.165-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:52.219-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:52.219-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:52.219-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:07:52.748-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:52.802-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:52.857-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:52.857-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:52.857-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:07:53.413-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:53.468-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:53.498-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:53.498-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:53.498-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:07:53.498-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:07:55.527-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:55.581-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:55.636-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:55.636-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:55.636-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:07:56.192-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:56.249-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:56.276-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:56.276-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:56.276-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:07:56.805-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:56.861-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:56.917-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:56.917-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:56.917-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:07:57.474-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:57.503-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:57.557-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:57.557-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:57.557-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:07:58.114-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:58.143-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:58.197-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:58.197-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:58.197-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:07:58.752-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:58.808-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:58.835-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:58.835-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:58.835-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:07:59.391-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:07:59.418-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:07:59.473-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:07:59.473-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:07:59.473-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:07:59.972-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:00.002-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:00.029-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:00.029-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:08:00.029-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:08:00.558-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:00.616-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:00.672-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:00.672-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:00.672-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:08:01.201-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:01.258-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:01.314-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:01.314-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:01.314-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:08:01.869-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:01.925-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:01.952-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:01.952-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:01.952-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:08:01.952-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:02.303-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:03.953-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:05.953-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:07.953-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:09.953-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:10.029-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:08:11.953-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:08:11.954-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:13.981-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:14.037-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:14.092-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:14.092-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:14.092-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:08:14.650-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:14.705-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:14.732-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:14.733-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:14.733-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:08:15.290-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:15.319-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:15.375-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:15.375-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:15.375-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:08:15.930-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:15.986-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:16.013-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:16.013-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:16.013-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:08:16.542-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:16.600-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:16.655-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:16.655-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:16.655-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:08:17.184-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:17.242-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:17.297-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:17.297-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:17.297-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:08:17.851-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:17.905-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:17.932-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:17.932-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:17.932-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:08:18.488-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:18.541-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:18.569-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:18.569-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:18.569-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:08:19.126-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:19.155-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:19.211-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:19.211-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:19.211-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:08:19.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:19.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:19.851-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:19.851-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:19.851-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:08:20.085-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:20.142-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:20.170-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:20.170-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:08:20.170-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:08:20.405-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:20.462-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:20.491-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:20.491-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:20.491-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:08:21.047-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:21.074-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:21.128-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:21.128-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:21.128-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:08:21.657-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:21.711-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:21.764-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:21.764-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:21.764-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:08:22.320-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:22.376-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:22.403-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:22.403-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:22.403-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:08:22.959-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:23.016-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:23.045-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:23.045-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:23.045-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:08:23.573-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:23.626-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:23.684-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:23.684-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:23.684-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:08:24.237-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:24.291-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:24.318-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:24.318-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:24.318-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:08:24.876-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:24.905-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:24.960-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:24.960-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:24.960-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:08:24.960-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:08:27.018-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:27.072-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:27.101-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:27.101-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:27.101-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:08:27.658-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:27.714-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:27.743-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:27.743-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:27.743-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:08:28.272-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:28.328-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:28.383-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:28.383-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:28.383-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:08:28.939-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:28.994-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:29.022-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:29.022-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:29.022-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:08:29.577-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:29.633-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:29.660-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:29.660-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:29.660-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:08:30.214-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:30.223-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:30.243-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:30.243-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:30.243-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:08:30.799-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:30.828-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:30.881-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:30.881-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:30.881-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:08:31.409-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:31.464-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:31.522-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:31.522-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:31.522-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:08:32.077-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:32.135-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:32.164-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:32.164-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:32.164-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:08:32.361-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:32.418-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:32.447-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:32.447-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:08:32.447-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:08:32.719-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:32.746-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:32.802-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:32.802-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:32.802-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:08:32.802-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:32.947-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:34.802-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:36.803-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:38.803-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:40.243-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:08:40.803-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:08:40.803-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:08:42.859-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:42.888-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:42.945-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:42.945-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:42.945-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:08:43.501-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:43.530-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:43.584-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:43.584-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:43.584-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:08:44.137-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:44.191-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:44.220-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:44.220-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:44.220-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:08:44.748-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:44.803-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:44.859-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:44.859-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:44.859-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:08:45.388-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:45.444-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:45.499-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:45.499-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:45.499-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:08:46.055-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:46.082-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:46.136-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:46.136-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:46.136-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:08:46.663-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:46.719-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:46.772-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:46.773-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:46.773-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:08:47.302-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:47.357-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:47.413-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:47.413-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:47.413-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:08:47.970-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:47.998-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:48.051-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:48.051-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:48.051-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:08:48.605-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:48.633-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:48.688-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:48.688-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:48.688-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:08:49.244-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:49.300-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:49.329-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:49.329-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:49.329-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:08:49.884-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:49.911-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:49.967-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:49.967-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:49.967-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:08:50.272-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:50.326-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:50.381-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:50.382-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:08:50.382-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:08:50.521-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:50.576-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:50.605-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:50.605-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:50.605-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:08:51.163-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:51.192-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:51.246-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:51.246-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:51.246-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:08:51.775-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:51.832-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:51.886-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:51.886-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:51.886-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:08:52.415-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:52.473-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:52.528-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:52.529-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:52.529-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:08:53.086-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:53.140-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:53.167-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:53.167-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:53.167-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:08:53.696-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:53.753-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:53.809-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:53.809-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:53.809-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:08:53.809-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:08:55.865-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:55.894-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:55.949-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:55.949-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:55.949-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:08:56.507-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:56.534-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:56.590-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:56.590-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:56.590-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:08:57.145-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:57.174-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:57.230-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:57.230-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:57.230-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:08:57.759-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:57.815-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:57.872-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:57.872-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:57.872-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:08:58.428-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:58.485-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:58.512-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:58.512-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:58.512-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:08:59.066-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:59.093-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:59.151-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:59.151-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:59.151-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:08:59.709-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:08:59.766-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:08:59.793-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:08:59.793-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:08:59.793-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:09:00.349-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:00.378-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:00.434-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:00.434-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:00.434-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:09:00.963-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:01.018-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:01.076-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:01.076-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:01.076-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:09:01.603-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:01.659-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:01.714-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:01.714-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:01.714-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:09:02.270-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:02.326-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:02.353-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:02.353-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:02.353-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:09:02.353-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:02.948-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:04.353-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:06.353-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:08.353-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:10.354-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:10.434-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:09:12.354-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:09:12.354-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:14.408-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:14.435-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:14.492-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:14.492-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:14.493-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:09:15.046-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:15.100-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:15.129-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:15.129-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:15.129-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:09:15.685-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:15.740-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:15.768-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:15.768-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:15.768-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:09:16.295-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:16.350-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:16.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:16.406-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:16.406-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:09:16.962-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:17.019-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:17.048-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:17.048-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:17.048-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:09:17.577-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:17.635-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:17.688-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:17.688-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:17.688-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:09:18.244-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:18.298-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:18.325-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:18.325-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:18.325-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:09:18.883-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:18.910-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:18.965-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:18.965-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:18.965-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:09:19.493-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:19.548-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:19.602-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:19.602-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:19.602-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:09:20.130-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:20.183-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:20.241-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:20.241-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:20.241-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:09:20.488-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:20.515-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:20.572-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:20.572-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:09:20.572-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:09:20.795-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:20.824-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:20.879-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:20.879-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:20.879-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:09:21.433-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:21.462-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:21.516-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:21.516-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:21.516-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:09:22.071-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:22.127-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:22.154-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:22.154-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:22.154-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:09:22.710-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:22.737-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:22.793-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:22.793-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:22.793-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:09:23.350-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:23.406-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:23.433-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:23.433-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:23.433-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:09:23.989-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:24.046-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:24.075-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:24.075-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:24.075-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:09:24.629-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:24.656-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:24.714-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:24.714-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:24.714-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:09:25.269-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:25.297-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:25.354-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:25.354-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:25.354-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:09:25.354-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:09:27.410-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:27.466-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:27.495-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:27.495-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:27.495-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:09:28.050-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:28.079-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:28.137-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:28.137-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:28.137-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:09:28.692-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:28.722-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:28.777-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:28.777-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:28.777-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:09:29.333-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:29.360-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:29.417-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:29.417-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:29.417-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:09:29.945-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:29.999-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:30.056-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:30.056-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:30.056-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:09:30.601-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:30.612-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:30.655-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:30.655-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:09:30.655-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:09:31.209-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:31.236-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:31.289-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:31.289-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:31.289-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:09:31.843-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:31.872-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:31.928-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:31.928-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:31.928-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:09:32.484-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:32.513-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:32.566-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:32.566-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:32.566-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:09:32.975-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:33.030-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:33.086-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:33.086-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:09:33.086-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:09:33.642-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:33.642-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:33.671-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:33.671-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:33.671-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:09:33.671-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:33.671-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:35.671-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:37.671-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:39.671-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:40.655-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:09:41.672-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:09:41.672-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:09:43.727-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:43.756-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:43.814-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:43.814-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:43.814-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:09:44.343-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:44.397-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:44.452-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:44.452-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:44.452-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:09:44.981-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:45.037-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:45.091-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:45.091-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:45.091-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:09:45.620-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:45.675-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:45.731-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:45.731-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:45.731-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:09:46.288-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:46.342-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:46.371-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:46.371-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:46.371-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:09:46.900-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:46.956-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:47.013-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:47.013-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:47.013-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:09:47.569-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:47.622-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:47.650-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:47.650-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:47.650-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:09:48.205-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:48.232-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:48.290-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:48.290-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:48.290-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:09:48.845-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:48.874-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:48.932-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:48.932-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:48.932-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:09:49.489-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:49.519-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:49.576-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:49.576-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:49.576-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:09:50.132-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:50.187-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:50.214-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:50.214-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:50.214-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:09:50.682-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:50.740-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:50.770-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:50.770-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:50.770-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:09:51.325-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:51.353-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:51.408-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:51.408-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:51.408-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:09:51.936-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:51.989-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:52.043-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:52.043-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:52.043-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:09:52.572-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:52.628-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:52.683-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:52.683-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:52.683-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:09:53.241-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:53.268-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:53.324-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:53.324-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:53.324-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:09:53.853-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:53.910-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:53.964-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:53.964-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:53.964-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:09:54.521-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:54.575-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:54.602-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:54.602-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:54.602-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:09:54.602-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:09:56.632-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:56.689-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:56.747-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:56.747-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:56.747-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:09:57.302-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:57.358-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:57.385-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:57.385-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:57.385-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:09:57.939-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:57.968-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:58.024-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:58.024-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:58.024-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:09:58.551-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:58.606-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:58.664-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:58.664-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:58.664-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:09:59.221-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:59.251-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:59.308-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:59.308-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:59.308-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:09:59.864-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:09:59.918-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:09:59.947-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:09:59.947-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:09:59.947-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:10:00.474-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:00.530-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:00.585-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:00.585-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:00.585-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:10:00.826-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:00.881-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:00.910-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:00.910-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:10:00.910-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:10:01.113-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:01.166-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:01.220-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:01.220-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:01.220-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:10:01.776-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:01.833-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:01.860-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:01.860-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:01.860-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:10:02.418-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:02.475-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:02.502-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:02.503-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:02.503-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:10:03.056-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:03.110-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:03.139-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:03.139-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:03.139-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:10:03.139-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:03.671-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:05.139-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:07.140-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:09.140-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:10.910-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:10:11.140-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:10:11.140-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:13.196-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:13.251-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:13.279-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:13.279-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:13.279-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:10:13.836-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:13.865-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:13.921-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:13.921-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:13.921-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:10:14.475-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:14.530-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:14.559-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:14.559-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:14.559-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:10:15.115-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:15.144-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:15.201-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:15.201-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:15.201-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:10:15.759-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:15.816-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:15.843-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:15.844-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:15.844-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:10:16.371-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:16.426-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:16.484-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:16.484-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:16.484-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:10:17.013-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:17.066-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:17.122-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:17.122-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:17.122-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:10:17.676-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:17.733-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:17.760-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:17.760-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:17.760-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:10:18.316-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:18.372-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:18.401-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:18.401-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:18.401-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:10:18.958-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:18.987-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:19.041-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:19.041-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:19.041-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:10:19.596-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:19.654-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:19.681-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:19.681-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:19.681-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:10:20.239-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:20.292-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:20.319-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:20.319-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:20.319-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:10:20.873-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:20.930-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:20.937-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:20.938-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:10:20.938-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:10:21.465-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:21.521-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:21.576-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:21.576-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:21.576-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:10:22.134-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:22.161-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:22.216-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:22.216-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:22.216-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:10:22.774-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:22.803-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:22.857-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:22.857-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:22.857-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:10:23.414-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:23.470-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:23.497-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:23.497-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:23.497-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:10:24.025-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:24.078-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:24.134-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:24.134-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:24.134-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:10:24.134-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:10:26.163-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:26.219-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:26.274-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:26.274-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:26.274-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:10:26.828-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:26.855-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:26.913-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:26.913-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:26.913-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:10:27.442-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:27.496-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:27.553-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:27.553-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:27.553-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:10:28.082-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:28.138-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:28.193-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:28.193-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:28.193-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:10:28.747-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:28.774-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:28.832-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:28.832-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:28.832-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:10:29.361-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:29.419-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:29.472-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:29.472-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:29.472-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:10:30.028-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:30.057-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:30.111-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:30.111-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:30.111-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:10:30.668-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:30.697-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:30.753-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:30.753-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:30.753-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:10:30.967-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:31.024-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:31.080-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:31.080-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:10:31.080-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:10:31.280-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:31.336-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:31.393-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:31.394-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:31.394-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:10:31.949-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:32.005-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:32.032-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:32.032-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:32.032-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:10:32.561-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:32.617-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:32.672-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:32.672-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:32.672-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:10:32.672-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:33.671-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:34.673-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:36.673-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:38.673-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:40.673-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:41.080-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:10:42.673-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:10:42.674-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:10:44.727-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:44.781-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:44.808-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:44.809-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:44.809-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:10:45.364-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:45.420-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:45.449-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:45.449-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:45.449-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:10:46.006-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:46.060-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:46.089-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:46.089-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:46.089-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:10:46.645-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:46.674-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:46.728-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:46.728-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:46.728-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:10:47.255-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:47.312-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:47.368-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:47.368-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:47.368-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:10:47.895-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:47.950-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:48.006-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:48.006-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:48.006-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:10:48.562-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:48.615-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:48.643-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:48.643-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:48.643-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:10:49.196-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:49.252-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:49.281-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:49.281-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:49.281-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:10:49.837-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:49.864-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:49.921-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:49.921-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:49.921-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:10:50.451-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:50.504-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:50.558-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:50.558-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:50.558-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:10:51.116-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:51.135-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:51.145-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:51.145-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:51.145-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:10:51.699-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:51.726-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:51.780-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:51.780-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:51.780-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:10:52.336-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:52.391-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:52.420-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:52.420-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:52.420-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:10:52.948-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:53.003-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:53.061-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:53.061-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:53.061-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:10:53.614-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:53.672-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:53.701-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:53.701-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:53.701-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:10:54.257-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:54.311-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:54.338-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:54.338-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:54.338-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:10:54.891-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:54.947-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:54.976-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:54.976-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:54.976-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:10:55.532-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:55.587-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:55.615-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:55.615-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:55.615-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:10:55.615-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:10:57.671-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:57.698-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:57.753-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:57.754-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:57.754-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:10:58.309-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:58.338-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:58.392-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:58.392-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:58.392-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:10:58.949-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:59.007-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:59.034-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:59.034-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:59.034-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:10:59.588-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:10:59.615-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:10:59.673-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:10:59.673-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:10:59.673-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:11:00.202-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:00.257-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:00.315-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:00.315-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:00.315-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:11:00.844-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:00.900-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:00.955-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:00.955-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:00.955-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:11:01.200-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:01.228-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:01.283-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:01.283-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:11:01.283-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:11:01.513-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:01.540-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:01.596-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:01.596-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:01.596-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:11:02.151-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:02.178-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:02.234-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:02.234-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:02.234-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:11:02.790-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:02.819-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:02.875-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:02.875-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:02.875-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:11:03.432-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:03.461-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:03.519-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:03.519-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:03.519-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:11:03.725-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:03.780-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:03.808-0500 W NETWORK [replSetDistLockPinger] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:03.808-0500 W NETWORK [replSetDistLockPinger] No primary detected for set csReplSet 2016-11-16T14:11:03.808-0500 I NETWORK [replSetDistLockPinger] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:11:03.808-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:04.019-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:06.019-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:08.019-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:10.020-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:11.283-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:11:12.020-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:11:12.020-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:14.074-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:14.129-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:14.158-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:14.159-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:14.159-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:11:14.688-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:14.743-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:14.799-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:14.799-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:14.799-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:11:15.356-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:15.383-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:15.439-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:15.439-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:15.439-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:11:15.995-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:16.024-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:16.078-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:16.078-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:16.078-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks 2016-11-16T14:11:16.632-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:16.687-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:16.714-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:16.714-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:16.714-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 5 checks in a row. Polling will stop after 25 more failed checks 2016-11-16T14:11:17.270-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:17.324-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:17.353-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:17.353-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:17.353-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 6 checks in a row. Polling will stop after 24 more failed checks 2016-11-16T14:11:17.909-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:17.938-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:17.993-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:17.993-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:17.993-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 7 checks in a row. Polling will stop after 23 more failed checks 2016-11-16T14:11:18.551-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:18.580-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:18.634-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:18.634-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:18.634-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 8 checks in a row. Polling will stop after 22 more failed checks 2016-11-16T14:11:19.189-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:19.247-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:19.276-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:19.276-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:19.276-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 9 checks in a row. Polling will stop after 21 more failed checks 2016-11-16T14:11:19.803-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:19.858-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:19.914-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:19.914-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:19.914-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 10 checks in a row. Polling will stop after 20 more failed checks 2016-11-16T14:11:20.443-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:20.499-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:20.552-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:20.552-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:20.552-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 11 checks in a row. Polling will stop after 19 more failed checks 2016-11-16T14:11:21.110-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:21.137-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:21.193-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:21.193-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:21.193-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 12 checks in a row. Polling will stop after 18 more failed checks 2016-11-16T14:11:21.311-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:21.366-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:21.422-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:21.422-0500 W NETWORK [ReplicaSetMonitorWatcher] No primary detected for set csReplSet 2016-11-16T14:11:21.422-0500 I NETWORK [ReplicaSetMonitorWatcher] All nodes for set csReplSet are down. This has happened for 13 checks in a row. Polling will stop after 17 more failed checks 2016-11-16T14:11:21.722-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:21.776-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:21.831-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:21.831-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:21.831-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 14 checks in a row. Polling will stop after 16 more failed checks 2016-11-16T14:11:22.389-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:22.416-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:22.473-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:22.473-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:22.473-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 15 checks in a row. Polling will stop after 15 more failed checks 2016-11-16T14:11:23.029-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:23.058-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:23.112-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:23.112-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:23.112-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 16 checks in a row. Polling will stop after 14 more failed checks 2016-11-16T14:11:23.667-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:23.696-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:23.754-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:23.754-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:23.754-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 17 checks in a row. Polling will stop after 13 more failed checks 2016-11-16T14:11:24.281-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:24.337-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:24.392-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:24.392-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:24.392-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 18 checks in a row. Polling will stop after 12 more failed checks 2016-11-16T14:11:24.921-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:24.977-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:25.032-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:25.032-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:25.032-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 19 checks in a row. Polling will stop after 11 more failed checks 2016-11-16T14:11:25.032-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: FailedToSatisfyReadPreference: could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csReplSet 2016-11-16T14:11:27.086-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:27.142-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:27.171-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:27.171-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:27.171-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 20 checks in a row. Polling will stop after 10 more failed checks 2016-11-16T14:11:27.698-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:27.753-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:27.811-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:27.811-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:27.811-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 21 checks in a row. Polling will stop after 9 more failed checks 2016-11-16T14:11:28.367-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:28.422-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:28.449-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:28.449-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:28.450-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 22 checks in a row. Polling will stop after 8 more failed checks 2016-11-16T14:11:29.007-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:29.036-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:29.091-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:29.092-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:29.092-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 23 checks in a row. Polling will stop after 7 more failed checks 2016-11-16T14:11:29.647-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:29.701-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:29.730-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:29.730-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:29.730-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 24 checks in a row. Polling will stop after 6 more failed checks 2016-11-16T14:11:30.286-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:30.341-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:30.370-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:30.370-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:30.370-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 25 checks in a row. Polling will stop after 5 more failed checks 2016-11-16T14:11:30.899-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:30.957-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:31.012-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:31.012-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:31.012-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 26 checks in a row. Polling will stop after 4 more failed checks 2016-11-16T14:11:31.479-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:31.537-0500 W NETWORK [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:31.541-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:31.541-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:31.541-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 27 checks in a row. Polling will stop after 3 more failed checks 2016-11-16T14:11:32.071-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:32.126-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:32.182-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:32.182-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:32.182-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 28 checks in a row. Polling will stop after 2 more failed checks 2016-11-16T14:11:32.737-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:32.766-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:32.824-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:32.824-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:32.824-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 29 checks in a row. Polling will stop after 1 more failed checks 2016-11-16T14:11:33.379-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:33.406-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:33.460-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:33.460-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:33.460-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 30 checks in a row. Polling will stop after 0 more failed checks 2016-11-16T14:11:33.460-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:33.808-0500 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: ReplicaSetNotFound: None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:35.461-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:37.461-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:39.461-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:41.461-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:41.542-0500 I NETWORK [ReplicaSetMonitorWatcher] Stopping periodic monitoring of set csReplSet because none of the hosts could be contacted for an extended period of time. 2016-11-16T14:11:43.461-0500 I NETWORK [mongosMain] Starting new replica set monitor for csReplSet/bxb-ppe-oas002:29102,bxb-ppe-oas012:29102,chi-ppe-oas019:29102 2016-11-16T14:11:43.461-0500 I SHARDING [mongosMain] Error initializing sharding state, sleeping for 2 seconds and trying again :: caused by :: ReplicaSetNotFound: could not get updated shard list from config server due to None of the hosts for replica set csReplSet could be contacted. 2016-11-16T14:11:45.515-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:45.543-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:45.600-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:45.600-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:45.600-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 1 checks in a row. Polling will stop after 29 more failed checks 2016-11-16T14:11:46.156-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:46.185-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:46.242-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:46.242-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:46.242-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 2 checks in a row. Polling will stop after 28 more failed checks 2016-11-16T14:11:46.769-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:46.825-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:46.882-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:46.882-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:46.882-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 3 checks in a row. Polling will stop after 27 more failed checks 2016-11-16T14:11:47.412-0500 W NETWORK [mongosMain] Failed to connect to 10.174.247.47:29102, reason: errno:111 Connection refused 2016-11-16T14:11:47.467-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.6:29102, reason: errno:111 Connection refused 2016-11-16T14:11:47.521-0500 W NETWORK [mongosMain] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused 2016-11-16T14:11:47.521-0500 W NETWORK [mongosMain] No primary detected for set csReplSet 2016-11-16T14:11:47.521-0500 I NETWORK [mongosMain] All nodes for set csReplSet are down. This has happened for 4 checks in a row. Polling will stop after 26 more failed checks