2016-11-02T14:09:31.145-0400 I CONTROL [initandlisten] MongoDB starting : pid=12055 port=27000 dbpath=shard0 64-bit host=liquid.local 2016-11-02T14:09:31.145-0400 I CONTROL [initandlisten] db version v3.4.0-rc2 2016-11-02T14:09:31.145-0400 I CONTROL [initandlisten] git version: 217fb0a4d2b808fa45921e4ca1d73db32330620b 2016-11-02T14:09:31.145-0400 I CONTROL [initandlisten] allocator: system 2016-11-02T14:09:31.145-0400 I CONTROL [initandlisten] modules: none 2016-11-02T14:09:31.145-0400 I CONTROL [initandlisten] build environment: 2016-11-02T14:09:31.145-0400 I CONTROL [initandlisten] distarch: x86_64 2016-11-02T14:09:31.145-0400 I CONTROL [initandlisten] target_arch: x86_64 2016-11-02T14:09:31.145-0400 I CONTROL [initandlisten] options: { net: { port: 27000 }, replication: { replSet: "rs0" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "shard0" }, systemLog: { destination: "file", path: "shard0.log" } } 2016-11-02T14:09:31.146-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=15872M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2016-11-02T14:09:32.120-0400 I CONTROL [initandlisten] 2016-11-02T14:09:32.120-0400 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2016-11-02T14:09:32.121-0400 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2016-11-02T14:09:32.121-0400 I CONTROL [initandlisten] 2016-11-02T14:09:32.121-0400 W SHARDING [initandlisten] Started with --shardsvr, but no shardIdentity document was found on disk in admin.system.version. This most likely means this server has not yet been added to a sharded cluster. 2016-11-02T14:09:32.336-0400 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory 'shard0/diagnostic.data' 2016-11-02T14:09:32.566-0400 I REPL [initandlisten] Did not find local voted for document at startup; NoMatchingDocument: Did not find replica set lastVote document in local.replset.election 2016-11-02T14:09:32.566-0400 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset 2016-11-02T14:09:32.566-0400 I NETWORK [thread1] waiting for connections on port 27000 2016-11-02T14:09:46.066-0400 I NETWORK [thread1] connection accepted from 127.0.0.1:59076 #1 (1 connection now open) 2016-11-02T14:09:48.515-0400 I COMMAND [conn1] initiate : no configuration specified. Using a default configuration for the set 2016-11-02T14:09:48.515-0400 I COMMAND [conn1] created this configuration for initiation : { _id: "rs0", version: 1, members: [ { _id: 0, host: "liquid.local:27000" } ] } 2016-11-02T14:09:48.515-0400 I REPL [conn1] replSetInitiate admin command received from client 2016-11-02T14:09:48.711-0400 I REPL [conn1] replSetInitiate config object with 1 members parses ok 2016-11-02T14:09:48.711-0400 I REPL [conn1] ****** 2016-11-02T14:09:48.711-0400 I REPL [conn1] creating replication oplog of size: 192MB... 2016-11-02T14:09:48.733-0400 I STORAGE [conn1] Starting WiredTigerRecordStoreThread local.oplog.rs 2016-11-02T14:09:48.733-0400 I STORAGE [conn1] The size storer reports that the oplog contains 0 records totaling to 0 bytes 2016-11-02T14:09:48.733-0400 I STORAGE [conn1] Scanning the oplog to determine where to place markers for truncation 2016-11-02T14:09:48.828-0400 I REPL [conn1] ****** 2016-11-02T14:09:48.955-0400 I REPL [conn1] New replica set config in use: { _id: "rs0", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "liquid.local:27000", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('581a2bec67693d643e3c065c') } } 2016-11-02T14:09:48.956-0400 I REPL [conn1] This node is liquid.local:27000 in the config 2016-11-02T14:09:48.956-0400 I REPL [conn1] transition to STARTUP2 2016-11-02T14:09:48.956-0400 I REPL [conn1] Starting replication storage threads 2016-11-02T14:09:48.956-0400 I REPL [conn1] Starting replication fetcher thread 2016-11-02T14:09:48.956-0400 I REPL [conn1] Starting replication applier thread 2016-11-02T14:09:48.956-0400 I REPL [conn1] Starting replication reporter thread 2016-11-02T14:09:48.956-0400 I COMMAND [conn1] command local.replset.minvalid command: replSetInitiate { replSetInitiate: undefined } numYields:0 reslen:130 locks:{ Global: { acquireCount: { r: 11, w: 5, W: 2 }, acquireWaitCount: { W: 1 }, timeAcquiringMicros: { W: 22 } }, Database: { acquireCount: { r: 2, w: 2, W: 3 } }, Collection: { acquireCount: { r: 2 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 2 } } } protocol:op_command 440ms 2016-11-02T14:09:48.956-0400 I REPL [rsSync] transition to RECOVERING 2016-11-02T14:09:48.956-0400 I REPL [rsSync] transition to SECONDARY 2016-11-02T14:09:48.956-0400 I REPL [rsSync] conducting a dry run election to see if we could be elected 2016-11-02T14:09:48.956-0400 I REPL [ReplicationExecutor] dry election run succeeded, running for election 2016-11-02T14:09:49.010-0400 I REPL [ReplicationExecutor] election succeeded, assuming primary role in term 1 2016-11-02T14:09:49.010-0400 I REPL [ReplicationExecutor] transition to PRIMARY 2016-11-02T14:09:49.010-0400 I REPL [ReplicationExecutor] Could not access any nodes within timeout when checking for additional ops to apply before finishing transition to primary. Will move forward with becoming primary anyway. 2016-11-02T14:09:49.959-0400 I REPL [rsSync] transition to primary complete; database writes are now permitted 2016-11-02T14:09:51.141-0400 I - [conn1] end connection 127.0.0.1:59076 (1 connection now open) 2016-11-02T14:11:15.273-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59098 #2 (1 connection now open) 2016-11-02T14:11:15.273-0400 I NETWORK [conn2] received client metadata from 10.0.1.3:59098 conn2: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:15.275-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59100 #3 (2 connections now open) 2016-11-02T14:11:15.275-0400 I NETWORK [conn3] received client metadata from 10.0.1.3:59100 conn3: { driver: { name: "AddShard-TaskExecutor", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:15.357-0400 I INDEX [conn3] build index on: admin.system.version properties: { v: 2, key: { version: 1 }, name: "incompatible_with_version_32", ns: "admin.system.version" } 2016-11-02T14:11:15.357-0400 I INDEX [conn3] building index using bulk method 2016-11-02T14:11:15.366-0400 I INDEX [conn3] build index done. scanned 0 total records. 0 secs 2016-11-02T14:11:15.367-0400 I COMMAND [conn3] setting featureCompatibilityVersion to 3.4 2016-11-02T14:11:15.374-0400 I SHARDING [conn3] initializing sharding state with: { _id: "shardIdentity", configsvrConnectionString: "csrs/liquid.local:27002", shardName: "rs0", clusterId: ObjectId('581a2bfba91ca21698bb6b2b') } 2016-11-02T14:11:15.374-0400 I SHARDING [conn3] first cluster operation detected, adding sharding hook to enable versioning and authentication to remote servers 2016-11-02T14:11:15.382-0400 I NETWORK [conn3] Starting new replica set monitor for csrs/liquid.local:27002 2016-11-02T14:11:15.382-0400 I SHARDING [thread2] creating distributed lock ping thread for process liquid.local:27000:1478110275:7476105823339711538 (sleeping for 30000ms) 2016-11-02T14:11:15.383-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27002 2016-11-02T14:11:15.384-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27002 2016-11-02T14:11:15.385-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to liquid.local:27002 2016-11-02T14:11:15.385-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to liquid.local:27002 2016-11-02T14:11:15.404-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document 2016-11-02T14:11:15.511-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59106 #4 (3 connections now open) 2016-11-02T14:11:15.511-0400 I NETWORK [conn4] received client metadata from 10.0.1.3:59106 conn4: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:20.119-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59133 #5 (4 connections now open) 2016-11-02T14:11:20.120-0400 I NETWORK [conn5] received client metadata from 10.0.1.3:59133 conn5: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:23.859-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59136 #6 (5 connections now open) 2016-11-02T14:11:23.860-0400 I NETWORK [conn6] received client metadata from 10.0.1.3:59136 conn6: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:23.928-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59140 #7 (6 connections now open) 2016-11-02T14:11:23.928-0400 I NETWORK [conn7] received client metadata from 10.0.1.3:59140 conn7: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:34.627-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59143 #8 (7 connections now open) 2016-11-02T14:11:34.627-0400 I NETWORK [conn8] received client metadata from 10.0.1.3:59143 conn8: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:39.649-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59145 #9 (8 connections now open) 2016-11-02T14:11:39.650-0400 I NETWORK [conn9] received client metadata from 10.0.1.3:59145 conn9: { driver: { name: "NetworkInterfaceASIO-TaskExecutorPool-0", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:45.385-0400 I NETWORK [shard registry reload] Starting new replica set monitor for rs0/liquid.local:27000 2016-11-02T14:11:45.385-0400 I NETWORK [shard registry reload] Starting new replica set monitor for rs1/liquid.local:27001 2016-11-02T14:11:45.386-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59146 #10 (9 connections now open) 2016-11-02T14:11:45.386-0400 I NETWORK [conn10] received client metadata from 10.0.1.3:59146 conn10: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:48.898-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59148 #11 (10 connections now open) 2016-11-02T14:11:48.898-0400 I NETWORK [conn11] received client metadata from 10.0.1.3:59148 conn11: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:48.929-0400 I INDEX [conn11] build index on: testDb.testColl properties: { v: 2, key: { b: 1.0, c: 1.0 }, name: "b_1_c_1", ns: "testDb.testColl" } 2016-11-02T14:11:48.929-0400 I INDEX [conn11] building index using bulk method 2016-11-02T14:11:48.948-0400 I INDEX [conn11] build index done. scanned 1000 total records. 0 secs 2016-11-02T14:11:57.095-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59151 #12 (11 connections now open) 2016-11-02T14:11:57.096-0400 I NETWORK [conn12] received client metadata from 10.0.1.3:59151 conn12: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:57.183-0400 I SHARDING [conn6] MetadataLoader loading chunks for testDb.testColl based on: (empty) 2016-11-02T14:11:57.184-0400 I SHARDING [conn6] Marking collection testDb.testColl as sharded with collection version: 1|0||581a2c6d172517c21c78fc70, shard version: 1|0||581a2c6d172517c21c78fc70 2016-11-02T14:11:57.184-0400 I SHARDING [conn6] MetadataLoader took 0 ms and found version 1|0||581a2c6d172517c21c78fc70 2016-11-02T14:12:43.979-0400 I SHARDING [conn7] received splitChunk request: { splitChunk: "testDb.testColl", configdb: "csrs/liquid.local:27002", from: "rs0", keyPattern: { b: 1.0, c: 1.0 }, shardVersion: [ Timestamp 1000|0, ObjectId('581a2c6d172517c21c78fc70') ], min: { b: MinKey, c: MinKey }, max: { b: MaxKey, c: MaxKey }, chunkVersion: [ Timestamp 1000|0, ObjectId('581a2c6d172517c21c78fc70') ], splitKeys: [ { b: 0.0, c: "a0" }, { b: 1000.0, c: "a1000" } ] } 2016-11-02T14:12:43.995-0400 I SHARDING [conn7] distributed lock 'testDb.testColl' acquired for 'splitting chunk [{ b: MinKey, c: MinKey }, { b: MaxKey, c: MaxKey }) in testDb.testColl', ts : 581a2c9b67693d643e3c0aaa 2016-11-02T14:12:43.995-0400 I SHARDING [conn7] MetadataLoader loading chunks for testDb.testColl based on: (empty) 2016-11-02T14:12:43.995-0400 I SHARDING [conn7] MetadataLoader took 0 ms and found version 1|0||581a2c6d172517c21c78fc70 2016-11-02T14:12:44.006-0400 I SHARDING [conn7] MetadataLoader loading chunks for testDb.testColl based on: (empty) 2016-11-02T14:12:44.007-0400 I SHARDING [conn7] Refreshing metadata for collection testDb.testColl from collection version: 1|0||581a2c6d172517c21c78fc70, shard version: 1|0||581a2c6d172517c21c78fc70 to collection version: 1|3||581a2c6d172517c21c78fc70, shard version: 1|3||581a2c6d172517c21c78fc70 2016-11-02T14:12:44.007-0400 I SHARDING [conn7] MetadataLoader took 0 ms and found version 1|3||581a2c6d172517c21c78fc70 2016-11-02T14:12:44.022-0400 I SHARDING [conn7] distributed lock with ts: 581a2c9b67693d643e3c0aaa' unlocked. 2016-11-02T14:12:44.029-0400 I - [conn7] end connection 10.0.1.3:59140 (11 connections now open) 2016-11-02T14:12:44.029-0400 I - [conn3] end connection 10.0.1.3:59100 (10 connections now open) 2016-11-02T14:12:44.029-0400 I - [conn2] end connection 10.0.1.3:59098 (10 connections now open) 2016-11-02T14:12:45.388-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Socket closed remotely, no longer connected (idle 13 secs, remote host 10.0.1.3:27002) 2016-11-02T14:12:45.388-0400 I ASIO [shard registry reload] dropping unhealthy pooled connection to liquid.local:27002 2016-11-02T14:12:45.388-0400 I ASIO [shard registry reload] after drop, pool was empty, going to spawn some connections 2016-11-02T14:12:45.388-0400 I ASIO [shard registry reload] Failed to close stream: Socket is not connected 2016-11-02T14:12:45.388-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27002 2016-11-02T14:12:45.389-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.389-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Marking host liquid.local:27002 as failed :: caused by :: Location13328: connection pool: connect failed liquid.local:27002 : couldn't connect to server liquid.local:27002, connection attempt failed 2016-11-02T14:12:45.389-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:12:45.389-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] All nodes for set csrs are down. This has happened for 1 checks in a row. 2016-11-02T14:12:45.389-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Failed to connect to liquid.local:27002 - HostUnreachable: Connection refused 2016-11-02T14:12:45.389-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.389-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:45.389-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 2 checks in a row. 2016-11-02T14:12:45.444-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.444-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:45.444-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 3 checks in a row. 2016-11-02T14:12:45.895-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.895-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:45.895-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 4 checks in a row. 2016-11-02T14:12:45.945-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:45.945-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:45.945-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 5 checks in a row. 2016-11-02T14:12:46.396-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.396-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:46.396-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 6 checks in a row. 2016-11-02T14:12:46.446-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.446-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:46.446-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 7 checks in a row. 2016-11-02T14:12:46.902-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.902-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:46.902-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 8 checks in a row. 2016-11-02T14:12:46.947-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:46.947-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:46.947-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 9 checks in a row. 2016-11-02T14:12:47.404-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.404-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:47.404-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 10 checks in a row. 2016-11-02T14:12:47.449-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.449-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:47.449-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 11 checks in a row. 2016-11-02T14:12:47.909-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.909-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:47.950-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:47.950-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:48.411-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.411-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:48.454-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.454-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:48.912-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.912-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:48.955-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:48.955-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:49.413-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.413-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:49.456-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.456-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:49.915-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.916-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:49.958-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:49.958-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:49.958-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 21 checks in a row. 2016-11-02T14:12:50.418-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.418-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:50.461-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.461-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:50.924-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.924-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:50.962-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.962-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:51.425-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.425-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:51.464-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.464-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:51.926-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.927-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:51.965-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.965-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:52.428-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.428-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:52.467-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.468-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:52.468-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 31 checks in a row. 2016-11-02T14:12:52.932-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.932-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:52.968-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.968-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:53.433-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.434-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:53.472-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.472-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:53.935-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.935-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:53.975-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.975-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:54.439-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.439-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:54.478-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.478-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:54.944-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.945-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:54.979-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.979-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:54.979-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 41 checks in a row. 2016-11-02T14:12:55.446-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.446-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:55.483-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.483-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:55.947-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.948-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:55.987-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.987-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:56.449-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.450-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:56.492-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.492-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:56.955-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.955-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:56.993-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.993-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:57.459-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.459-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:57.496-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.497-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:57.497-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 51 checks in a row. 2016-11-02T14:12:57.961-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.961-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:58.001-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.001-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:58.463-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.463-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:58.503-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.503-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:58.969-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.969-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:59.007-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.007-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:59.473-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.473-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:59.509-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.509-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:59.978-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.978-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:00.012-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.012-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:00.012-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 61 checks in a row. 2016-11-02T14:13:00.481-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.481-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:00.516-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.516-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:00.984-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.984-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:01.019-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.019-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:01.485-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.485-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:01.520-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.520-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:01.987-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.987-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:02.025-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.026-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:02.492-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.492-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:02.526-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.526-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:02.526-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 71 checks in a row. 2016-11-02T14:13:02.994-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.994-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:03.027-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.027-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:03.498-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.498-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:03.528-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.528-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:04.002-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.002-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:04.029-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.029-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:04.503-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.504-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:04.533-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.533-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:05.010-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.010-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:05.036-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.036-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:05.036-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 81 checks in a row. 2016-11-02T14:13:05.512-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.512-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:05.512-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:13:05.539-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.539-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:05.539-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:13:15.390-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:15.390-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:13:35.515-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:35.515-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:35.542-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:35.542-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:36.021-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.021-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:36.043-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.043-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:36.524-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.524-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:36.545-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:36.545-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:37.027-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.028-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:37.028-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 91 checks in a row. 2016-11-02T14:13:37.049-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.049-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:37.532-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.532-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:37.550-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:37.550-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:38.037-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.037-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:38.053-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.053-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:38.539-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.539-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:38.554-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:38.554-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:39.040-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.040-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:39.056-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.057-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:39.545-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.545-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:39.545-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 101 checks in a row. 2016-11-02T14:13:39.559-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:39.559-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:40.051-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.051-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:40.062-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.062-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:40.556-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.556-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:40.568-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.568-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:41.061-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.061-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:41.069-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.069-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:41.562-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.562-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:41.574-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.574-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:42.067-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.067-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:42.067-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 111 checks in a row. 2016-11-02T14:13:42.077-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.077-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:42.572-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.572-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:42.583-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.583-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:43.076-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.076-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:43.085-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.085-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:43.579-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.579-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:43.587-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.587-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:44.082-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.082-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:44.090-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.090-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:44.587-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.587-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:44.587-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 121 checks in a row. 2016-11-02T14:13:44.593-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.593-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:45.093-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.093-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:45.099-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.099-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:45.391-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.391-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:13:45.596-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.596-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:45.601-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.601-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:46.101-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.101-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:46.102-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.102-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:46.606-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.606-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:47.110-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.110-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:47.110-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 131 checks in a row. 2016-11-02T14:13:47.615-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.615-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:48.120-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.120-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:48.624-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.624-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:49.129-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.129-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:49.630-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.630-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:50.134-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.134-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:50.640-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.640-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:51.146-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.146-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:51.647-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.647-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:52.148-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.149-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:52.149-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 141 checks in a row. 2016-11-02T14:13:52.652-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.653-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:53.154-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.154-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:53.660-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.660-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:54.163-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.163-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:54.668-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.668-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:55.172-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.172-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:55.673-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.673-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:55.673-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:13:55.673-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:14:15.392-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:15.392-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:14:25.676-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:25.676-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:26.179-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:26.179-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:26.179-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 151 checks in a row. 2016-11-02T14:14:26.684-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:26.684-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:27.187-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:27.187-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:27.689-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:27.689-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:28.192-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:28.192-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:28.695-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:28.695-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:29.200-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:29.200-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:29.704-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:29.704-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:30.207-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.207-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:30.711-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.711-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:31.215-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.215-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:31.215-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 161 checks in a row. 2016-11-02T14:14:31.718-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.718-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:32.224-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.224-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:32.729-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.729-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:33.234-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.234-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:33.740-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.740-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:34.241-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.241-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:34.743-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.743-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:35.248-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.248-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:35.749-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.749-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:36.254-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.254-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:36.254-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 171 checks in a row. 2016-11-02T14:14:36.755-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.755-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:37.261-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.261-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:37.766-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.766-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:38.270-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.271-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:38.772-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.772-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:39.274-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.274-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:39.776-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.776-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:40.279-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.279-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:40.782-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.782-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:41.283-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.283-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:41.283-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 181 checks in a row. 2016-11-02T14:14:41.789-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.789-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:42.293-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.293-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:42.794-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.794-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:43.299-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.299-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:43.800-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.800-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:44.302-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.302-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:44.803-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.803-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:45.309-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.309-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:45.393-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.393-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:14:45.815-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.815-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:45.815-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 191 checks in a row. 2016-11-02T14:14:45.815-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:14:45.815-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:15:15.393-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:15.393-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:15:15.815-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:15.815-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:16.320-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:16.320-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:16.824-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:16.825-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:17.327-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:17.327-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:17.828-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:17.828-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:18.329-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:18.329-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:18.831-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:18.831-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:19.335-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:19.335-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:19.837-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:19.837-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:19.837-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 201 checks in a row. 2016-11-02T14:15:20.340-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.340-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:20.845-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.845-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:21.347-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.347-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:21.852-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.852-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:22.356-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.357-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:22.862-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.862-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:23.365-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.365-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:23.868-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.868-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:24.373-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.373-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:24.875-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.875-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:24.875-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 211 checks in a row. 2016-11-02T14:15:25.378-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.378-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:25.882-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.882-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:26.383-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.383-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:26.885-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.885-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:27.390-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.390-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:27.893-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.893-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:28.394-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.394-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:28.895-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.896-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:29.401-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.401-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:29.902-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.902-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:29.902-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 221 checks in a row. 2016-11-02T14:15:30.403-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.404-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:30.904-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.904-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:31.406-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.406-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:31.910-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.910-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:32.414-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.414-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:32.917-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.917-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:33.421-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.421-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:33.922-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.923-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:34.428-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.428-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:34.931-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.931-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:34.931-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 231 checks in a row. 2016-11-02T14:15:35.436-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.436-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:35.940-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.940-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:35.940-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:15:35.940-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:15:45.397-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:45.397-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:16:05.943-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:05.943-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:06.448-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:06.448-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:06.951-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:06.951-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:07.456-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:07.456-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:07.958-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:07.958-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:08.461-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:08.461-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:08.965-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:08.965-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:08.965-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 241 checks in a row. 2016-11-02T14:16:09.466-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:09.466-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:09.967-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:09.967-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:10.468-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:10.469-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:10.972-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:10.972-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:11.475-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.475-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:11.976-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.976-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:12.480-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.480-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:12.984-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.984-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:13.489-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.489-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:13.990-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.990-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:13.990-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 251 checks in a row. 2016-11-02T14:16:14.495-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.495-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:14.996-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.996-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:15.397-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.397-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:16:15.497-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.497-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:15.999-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.999-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:16.504-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.504-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:17.006-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.006-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:17.510-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.511-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:18.016-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.016-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:18.522-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.522-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:18.522-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 261 checks in a row. 2016-11-02T14:16:19.025-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.025-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:19.528-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.528-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:20.030-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.030-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:20.533-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.533-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:21.035-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.035-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:21.540-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.541-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:22.044-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.044-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:22.546-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.547-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:23.047-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.048-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:23.552-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.552-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:23.552-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 271 checks in a row. 2016-11-02T14:16:24.057-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.057-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:24.558-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.558-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:25.059-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.059-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:25.560-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.560-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:26.063-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.063-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:26.063-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:16:26.063-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:16:39.651-0400 I - [conn9] end connection 10.0.1.3:59145 (8 connections now open) 2016-11-02T14:16:45.397-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:45.397-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:16:56.063-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:56.063-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:56.068-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:56.068-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:56.567-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:56.567-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:56.568-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:56.568-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:56.568-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 281 checks in a row. 2016-11-02T14:16:57.071-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:57.071-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:57.186-0400 I - [conn6] end connection 10.0.1.3:59136 (7 connections now open) 2016-11-02T14:16:57.574-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:57.575-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:58.077-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:58.077-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:58.581-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:58.581-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:59.086-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:59.086-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:59.588-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:59.588-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:00.094-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.094-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:00.597-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.597-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:01.098-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.098-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:01.602-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.602-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:01.602-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 291 checks in a row. 2016-11-02T14:17:02.103-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.103-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:02.608-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.608-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:03.114-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.114-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:03.620-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.620-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:04.123-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.123-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:04.624-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.624-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:05.127-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.127-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:05.629-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.630-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:06.134-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.134-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:06.636-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.636-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:06.636-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 301 checks in a row. 2016-11-02T14:17:07.137-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.137-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:07.641-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.641-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:08.144-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.144-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:08.647-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.648-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:09.151-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.151-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:09.655-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.655-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:10.156-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.156-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:10.658-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.658-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:11.162-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.162-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:11.666-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.666-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:11.666-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 311 checks in a row. 2016-11-02T14:17:12.168-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.168-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:12.670-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.670-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:13.174-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.174-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:13.675-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.675-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:14.177-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.177-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:14.683-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.683-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:15.187-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.187-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:15.400-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.400-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:17:15.691-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.691-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:16.196-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.196-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:16.196-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 321 checks in a row. 2016-11-02T14:17:16.197-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:17:16.197-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:17:45.400-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:45.400-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:17:46.200-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:46.200-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:46.706-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:46.707-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:47.210-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:47.210-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:47.712-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:47.712-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:48.216-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:48.216-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:48.721-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:48.722-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:49.224-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:49.224-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:49.728-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:49.728-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:50.234-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.234-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:50.234-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 331 checks in a row. 2016-11-02T14:17:50.736-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.736-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:51.241-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.241-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:51.743-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.743-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:52.246-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.246-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:52.747-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.747-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:53.248-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.248-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:53.750-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.750-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:54.253-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.253-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:54.756-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.756-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:55.259-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.259-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:55.259-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 341 checks in a row. 2016-11-02T14:17:55.765-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.765-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:56.268-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.268-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:56.770-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.770-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:57.276-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.276-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:57.777-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.777-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:58.281-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.281-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:58.782-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.782-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:59.286-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.286-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:59.787-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.787-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:00.293-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.293-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:00.293-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 351 checks in a row. 2016-11-02T14:18:00.796-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.796-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:01.298-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.298-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:01.800-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.800-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:02.306-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.306-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:02.811-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.811-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:03.314-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.314-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:03.820-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.820-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:04.321-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.321-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:04.823-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.823-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:05.324-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.324-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:05.324-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 361 checks in a row. 2016-11-02T14:18:05.830-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.830-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:06.336-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:06.337-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:06.337-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:18:06.337-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:18:15.403-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:15.403-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:18:36.337-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:36.337-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:36.838-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:36.838-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:37.341-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:37.341-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:37.846-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:37.846-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:38.351-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:38.351-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:38.854-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:38.854-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:39.355-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:39.355-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:39.355-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 371 checks in a row. 2016-11-02T14:18:39.860-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:39.860-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:40.364-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:40.365-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:40.870-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:40.871-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:41.372-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.372-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:41.877-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.877-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:42.383-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.383-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:42.884-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.885-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:43.388-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.388-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:43.889-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.889-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:44.392-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.392-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:44.392-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 381 checks in a row. 2016-11-02T14:18:44.896-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.896-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:45.398-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.398-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:45.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.404-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:18:45.902-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.902-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:46.408-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.408-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:46.910-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.910-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:47.414-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.415-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:47.916-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.916-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:48.420-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.420-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:48.921-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.921-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:48.921-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 391 checks in a row. 2016-11-02T14:18:49.424-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.424-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:49.926-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.926-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:50.430-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.430-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:50.931-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.932-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:51.432-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.432-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:51.933-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.933-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:52.435-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.435-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:52.939-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.939-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:53.440-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.440-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:53.941-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.941-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:53.941-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 401 checks in a row. 2016-11-02T14:18:54.445-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.445-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:54.950-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.950-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:55.454-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.454-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:55.958-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.958-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:56.462-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:56.462-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:56.462-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:18:56.462-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:19:15.408-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:15.408-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:19:26.462-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:26.463-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:26.463-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:26.463-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:26.963-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:26.963-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:26.964-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:26.964-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:26.964-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 411 checks in a row. 2016-11-02T14:19:27.469-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:27.469-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:27.969-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:27.969-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:28.473-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:28.473-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:28.974-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:28.974-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:29.479-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:29.479-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:29.982-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:29.982-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:30.483-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:30.483-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:30.984-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:30.984-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:31.490-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:31.491-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:31.996-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:31.997-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:31.997-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 421 checks in a row. 2016-11-02T14:19:32.499-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:32.500-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:33.002-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:33.003-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:33.505-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:33.506-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:34.008-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:34.008-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:34.512-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:34.512-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:35.013-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:35.014-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:35.518-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:35.518-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:36.020-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:36.020-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:36.522-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:36.522-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:37.024-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:37.024-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:37.024-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 431 checks in a row. 2016-11-02T14:19:37.530-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:37.530-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:38.031-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:38.031-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:38.534-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:38.534-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:39.036-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:39.036-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:39.537-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:39.537-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:40.041-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:40.041-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:40.543-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:40.543-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:41.044-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:41.044-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:41.548-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:41.548-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:42.052-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:42.052-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:42.052-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 441 checks in a row. 2016-11-02T14:19:42.558-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:42.558-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:43.063-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:43.063-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:43.564-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:43.564-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:44.070-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:44.070-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:44.571-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:44.572-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:45.077-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:45.077-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:45.409-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:45.409-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:19:45.580-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:45.580-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:46.083-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:46.083-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:46.586-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:46.586-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:46.586-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 451 checks in a row. 2016-11-02T14:19:46.586-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:19:46.586-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:20:15.410-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:15.410-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:20:16.587-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:16.587-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:17.089-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:17.089-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:17.593-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:17.593-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:18.094-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:18.094-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:18.600-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:18.600-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:19.106-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:19.106-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:19.610-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:19.610-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:20.115-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:20.116-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:20.617-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:20.617-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:20.617-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 461 checks in a row. 2016-11-02T14:20:21.118-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.118-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:21.624-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.624-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:22.127-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.127-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:22.629-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.629-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:23.130-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.130-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:23.631-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.631-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:24.132-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.132-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:24.635-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.635-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:25.140-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.140-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:25.644-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.644-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:25.644-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 471 checks in a row. 2016-11-02T14:20:26.148-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.148-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:26.652-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.652-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:27.153-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.153-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:27.657-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.657-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:28.162-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.162-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:28.667-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.667-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:29.172-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.172-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:29.673-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.673-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:30.174-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:30.174-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:30.676-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:30.676-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:30.676-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 481 checks in a row. 2016-11-02T14:20:31.179-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.179-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:31.685-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.685-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:32.186-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.186-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:32.689-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.689-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:32.723-0400 I CONTROL [signalProcessingThread] got signal 2 (Interrupt: 2), will terminate after current cmd ends 2016-11-02T14:20:32.723-0400 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... 2016-11-02T14:20:32.723-0400 I NETWORK [signalProcessingThread] closing listening socket: 7 2016-11-02T14:20:32.723-0400 I NETWORK [signalProcessingThread] closing listening socket: 8 2016-11-02T14:20:32.724-0400 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-27000.sock 2016-11-02T14:20:32.726-0400 I NETWORK [signalProcessingThread] shutdown: going to flush diaglog... 2016-11-02T14:20:32.726-0400 I REPL [signalProcessingThread] shutting down replication subsystems 2016-11-02T14:20:32.726-0400 I REPL [signalProcessingThread] Stopping replication reporter thread 2016-11-02T14:20:32.726-0400 I REPL [signalProcessingThread] Stopping replication fetcher thread 2016-11-02T14:20:32.726-0400 I REPL [signalProcessingThread] Stopping replication applier thread 2016-11-02T14:20:33.192-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:33.269-0400 I REPL [signalProcessingThread] Stopping replication storage threads 2016-11-02T14:20:33.696-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: InterruptedAtShutdown: interrupted at shutdown 2016-11-02T14:20:33.696-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 11600 could not get updated shard list from config server due to interrupted at shutdown; will retry after 30s 2016-11-02T14:20:33.696-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture 2016-11-02T14:20:33.698-0400 W SHARDING [signalProcessingThread] error encountered while cleaning up distributed ping entry for liquid.local:27000:1478110275:7476105823339711538 :: caused by :: InterruptedAtShutdown: interrupted at shutdown 2016-11-02T14:20:33.698-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down 2016-11-02T14:20:33.821-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock... 2016-11-02T14:20:33.822-0400 I CONTROL [signalProcessingThread] now exiting 2016-11-02T14:20:33.822-0400 I CONTROL [signalProcessingThread] shutting down with code:0 2016-11-02T14:20:33.822-0400 I CONTROL [initandlisten] shutting down with code:0