2016-11-02T14:09:36.640-0400 I CONTROL [initandlisten] MongoDB starting : pid=12062 port=27001 dbpath=shard1 64-bit host=liquid.local 2016-11-02T14:09:36.640-0400 I CONTROL [initandlisten] db version v3.4.0-rc2 2016-11-02T14:09:36.640-0400 I CONTROL [initandlisten] git version: 217fb0a4d2b808fa45921e4ca1d73db32330620b 2016-11-02T14:09:36.640-0400 I CONTROL [initandlisten] allocator: system 2016-11-02T14:09:36.640-0400 I CONTROL [initandlisten] modules: none 2016-11-02T14:09:36.640-0400 I CONTROL [initandlisten] build environment: 2016-11-02T14:09:36.640-0400 I CONTROL [initandlisten] distarch: x86_64 2016-11-02T14:09:36.640-0400 I CONTROL [initandlisten] target_arch: x86_64 2016-11-02T14:09:36.640-0400 I CONTROL [initandlisten] options: { net: { port: 27001 }, replication: { replSet: "rs1" }, sharding: { clusterRole: "shardsvr" }, storage: { dbPath: "shard1" }, systemLog: { destination: "file", path: "shard1.log" } } 2016-11-02T14:09:36.641-0400 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=15872M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2016-11-02T14:09:37.525-0400 I CONTROL [initandlisten] 2016-11-02T14:09:37.525-0400 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2016-11-02T14:09:37.525-0400 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2016-11-02T14:09:37.525-0400 I CONTROL [initandlisten] 2016-11-02T14:09:37.525-0400 W SHARDING [initandlisten] Started with --shardsvr, but no shardIdentity document was found on disk in admin.system.version. This most likely means this server has not yet been added to a sharded cluster. 2016-11-02T14:09:37.756-0400 I FTDC [initandlisten] Initializing full-time diagnostic data capture with directory 'shard1/diagnostic.data' 2016-11-02T14:09:37.993-0400 I REPL [initandlisten] Did not find local voted for document at startup; NoMatchingDocument: Did not find replica set lastVote document in local.replset.election 2016-11-02T14:09:37.993-0400 I REPL [initandlisten] Did not find local replica set configuration document at startup; NoMatchingDocument: Did not find replica set configuration document in local.system.replset 2016-11-02T14:09:37.993-0400 I NETWORK [thread1] waiting for connections on port 27001 2016-11-02T14:09:52.220-0400 I NETWORK [thread1] connection accepted from 127.0.0.1:59077 #1 (1 connection now open) 2016-11-02T14:09:54.555-0400 I COMMAND [conn1] initiate : no configuration specified. Using a default configuration for the set 2016-11-02T14:09:54.555-0400 I COMMAND [conn1] created this configuration for initiation : { _id: "rs1", version: 1, members: [ { _id: 0, host: "liquid.local:27001" } ] } 2016-11-02T14:09:54.555-0400 I REPL [conn1] replSetInitiate admin command received from client 2016-11-02T14:09:54.556-0400 I REPL [conn1] replSetInitiate config object with 1 members parses ok 2016-11-02T14:09:54.556-0400 I REPL [conn1] ****** 2016-11-02T14:09:54.556-0400 I REPL [conn1] creating replication oplog of size: 192MB... 2016-11-02T14:09:54.597-0400 I STORAGE [conn1] Starting WiredTigerRecordStoreThread local.oplog.rs 2016-11-02T14:09:54.597-0400 I STORAGE [conn1] The size storer reports that the oplog contains 0 records totaling to 0 bytes 2016-11-02T14:09:54.597-0400 I STORAGE [conn1] Scanning the oplog to determine where to place markers for truncation 2016-11-02T14:09:54.745-0400 I REPL [conn1] ****** 2016-11-02T14:09:54.850-0400 I REPL [conn1] New replica set config in use: { _id: "rs1", version: 1, protocolVersion: 1, members: [ { _id: 0, host: "liquid.local:27001", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 2000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('581a2bf25e0ffbbe66dc890d') } } 2016-11-02T14:09:54.851-0400 I REPL [conn1] This node is liquid.local:27001 in the config 2016-11-02T14:09:54.851-0400 I REPL [conn1] transition to STARTUP2 2016-11-02T14:09:54.851-0400 I REPL [conn1] Starting replication storage threads 2016-11-02T14:09:54.851-0400 I REPL [conn1] Starting replication fetcher thread 2016-11-02T14:09:54.851-0400 I REPL [conn1] Starting replication applier thread 2016-11-02T14:09:54.851-0400 I REPL [conn1] Starting replication reporter thread 2016-11-02T14:09:54.851-0400 I COMMAND [conn1] command local.replset.minvalid command: replSetInitiate { replSetInitiate: undefined } numYields:0 reslen:130 locks:{ Global: { acquireCount: { r: 11, w: 5, W: 2 }, acquireWaitCount: { W: 1 }, timeAcquiringMicros: { W: 21 } }, Database: { acquireCount: { r: 2, w: 2, W: 3 } }, Collection: { acquireCount: { r: 2 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 2 } } } protocol:op_command 295ms 2016-11-02T14:09:54.851-0400 I REPL [rsSync] transition to RECOVERING 2016-11-02T14:09:54.852-0400 I REPL [rsSync] transition to SECONDARY 2016-11-02T14:09:54.852-0400 I REPL [rsSync] conducting a dry run election to see if we could be elected 2016-11-02T14:09:54.852-0400 I REPL [ReplicationExecutor] dry election run succeeded, running for election 2016-11-02T14:09:54.903-0400 I REPL [ReplicationExecutor] election succeeded, assuming primary role in term 1 2016-11-02T14:09:54.903-0400 I REPL [ReplicationExecutor] transition to PRIMARY 2016-11-02T14:09:54.903-0400 I REPL [ReplicationExecutor] Could not access any nodes within timeout when checking for additional ops to apply before finishing transition to primary. Will move forward with becoming primary anyway. 2016-11-02T14:09:55.852-0400 I REPL [rsSync] transition to primary complete; database writes are now permitted 2016-11-02T14:09:56.973-0400 I - [conn1] end connection 127.0.0.1:59077 (1 connection now open) 2016-11-02T14:11:20.003-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59125 #2 (1 connection now open) 2016-11-02T14:11:20.003-0400 I NETWORK [conn2] received client metadata from 10.0.1.3:59125 conn2: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:20.004-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59127 #3 (2 connections now open) 2016-11-02T14:11:20.004-0400 I NETWORK [conn3] received client metadata from 10.0.1.3:59127 conn3: { driver: { name: "AddShard-TaskExecutor", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:20.085-0400 I INDEX [conn3] build index on: admin.system.version properties: { v: 2, key: { version: 1 }, name: "incompatible_with_version_32", ns: "admin.system.version" } 2016-11-02T14:11:20.085-0400 I INDEX [conn3] building index using bulk method 2016-11-02T14:11:20.096-0400 I INDEX [conn3] build index done. scanned 0 total records. 0 secs 2016-11-02T14:11:20.097-0400 I COMMAND [conn3] setting featureCompatibilityVersion to 3.4 2016-11-02T14:11:20.105-0400 I COMMAND [conn3] command admin.$cmd command: setFeatureCompatibilityVersion { v: 2, key: { version: 1 }, ns: "admin.system.version", name: "incompatible_with_version_32" } numYields:0 reslen:22 locks:{ Global: { acquireCount: { r: 5, w: 5 } }, Database: { acquireCount: { w: 3, W: 2 } }, Collection: { acquireCount: { w: 2 } }, Metadata: { acquireCount: { w: 3 } }, oplog: { acquireCount: { w: 3 } } } protocol:op_command 100ms 2016-11-02T14:11:20.106-0400 I SHARDING [conn3] initializing sharding state with: { _id: "shardIdentity", configsvrConnectionString: "csrs/liquid.local:27002", shardName: "rs1", clusterId: ObjectId('581a2bfba91ca21698bb6b2b') } 2016-11-02T14:11:20.106-0400 I SHARDING [conn3] first cluster operation detected, adding sharding hook to enable versioning and authentication to remote servers 2016-11-02T14:11:20.115-0400 I NETWORK [conn3] Starting new replica set monitor for csrs/liquid.local:27002 2016-11-02T14:11:20.115-0400 I SHARDING [thread2] creating distributed lock ping thread for process liquid.local:27001:1478110280:3025486684967422951 (sleeping for 30000ms) 2016-11-02T14:11:20.117-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27002 2016-11-02T14:11:20.117-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27002 2016-11-02T14:11:20.119-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to liquid.local:27002 2016-11-02T14:11:20.119-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Successfully connected to liquid.local:27002 2016-11-02T14:11:20.119-0400 I NETWORK [shard registry reload] Starting new replica set monitor for rs0/liquid.local:27000 2016-11-02T14:11:20.135-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: LockStateChangeFailed: findAndModify query predicate didn't match any lock document 2016-11-02T14:11:20.173-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59134 #4 (3 connections now open) 2016-11-02T14:11:20.174-0400 I NETWORK [conn4] received client metadata from 10.0.1.3:59134 conn4: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:23.862-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59138 #5 (4 connections now open) 2016-11-02T14:11:23.863-0400 I NETWORK [conn5] received client metadata from 10.0.1.3:59138 conn5: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:23.929-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59142 #6 (5 connections now open) 2016-11-02T14:11:23.929-0400 I NETWORK [conn6] received client metadata from 10.0.1.3:59142 conn6: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:45.386-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59147 #7 (6 connections now open) 2016-11-02T14:11:45.387-0400 I NETWORK [conn7] received client metadata from 10.0.1.3:59147 conn7: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:11:50.120-0400 I NETWORK [shard registry reload] Starting new replica set monitor for rs1/liquid.local:27001 2016-11-02T14:11:50.121-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59149 #8 (7 connections now open) 2016-11-02T14:11:50.121-0400 I NETWORK [conn8] received client metadata from 10.0.1.3:59149 conn8: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:12:44.029-0400 I - [conn6] end connection 10.0.1.3:59142 (7 connections now open) 2016-11-02T14:12:44.029-0400 I - [conn2] end connection 10.0.1.3:59125 (7 connections now open) 2016-11-02T14:12:44.029-0400 I - [conn3] end connection 10.0.1.3:59127 (6 connections now open) 2016-11-02T14:12:50.121-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Socket closed remotely, no longer connected (idle 13 secs, remote host 10.0.1.3:27002) 2016-11-02T14:12:50.122-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.122-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] Marking host liquid.local:27002 as failed :: caused by :: Location13328: connection pool: connect failed liquid.local:27002 : couldn't connect to server liquid.local:27002, connection attempt failed 2016-11-02T14:12:50.122-0400 I ASIO [shard registry reload] dropping unhealthy pooled connection to liquid.local:27002 2016-11-02T14:12:50.122-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:12:50.122-0400 I ASIO [shard registry reload] after drop, pool was empty, going to spawn some connections 2016-11-02T14:12:50.122-0400 I NETWORK [ReplicaSetMonitor-TaskExecutor-0] All nodes for set csrs are down. This has happened for 1 checks in a row. 2016-11-02T14:12:50.122-0400 I ASIO [shard registry reload] Failed to close stream: Socket is not connected 2016-11-02T14:12:50.122-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Connecting to liquid.local:27002 2016-11-02T14:12:50.122-0400 I ASIO [NetworkInterfaceASIO-ShardRegistry-0] Failed to connect to liquid.local:27002 - HostUnreachable: Connection refused 2016-11-02T14:12:50.123-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.123-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:50.123-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 2 checks in a row. 2016-11-02T14:12:50.159-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.159-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:50.159-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 3 checks in a row. 2016-11-02T14:12:50.624-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.624-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:50.624-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 4 checks in a row. 2016-11-02T14:12:50.663-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:50.663-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:50.663-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 5 checks in a row. 2016-11-02T14:12:51.125-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.125-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:51.125-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 6 checks in a row. 2016-11-02T14:12:51.167-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.167-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:51.167-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 7 checks in a row. 2016-11-02T14:12:51.628-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.628-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:51.628-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 8 checks in a row. 2016-11-02T14:12:51.672-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:51.672-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:51.672-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 9 checks in a row. 2016-11-02T14:12:52.130-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.130-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:52.130-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 10 checks in a row. 2016-11-02T14:12:52.176-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.176-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:52.176-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 11 checks in a row. 2016-11-02T14:12:52.631-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.631-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:52.681-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:52.681-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:53.132-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.132-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:53.182-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.182-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:53.633-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.633-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:53.683-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:53.683-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:54.133-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.134-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:54.184-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.184-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:54.635-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.635-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:54.686-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:54.686-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:54.686-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 21 checks in a row. 2016-11-02T14:12:55.136-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.136-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:55.187-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.187-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:55.639-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.639-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:55.688-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:55.688-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:56.144-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.144-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:56.191-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.191-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:56.648-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.648-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:56.696-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:56.697-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:57.149-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.149-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:57.199-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.199-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:57.199-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 31 checks in a row. 2016-11-02T14:12:57.650-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.650-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:57.703-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:57.703-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:58.151-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.151-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:58.208-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.208-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:58.651-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.651-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:58.711-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:58.711-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:59.155-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.155-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:59.215-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.215-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:59.656-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.656-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:12:59.716-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:12:59.716-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:12:59.716-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 41 checks in a row. 2016-11-02T14:13:00.157-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.157-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:00.217-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.217-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:00.658-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.658-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:00.719-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:00.719-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:01.161-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.161-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:01.222-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.222-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:01.666-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.666-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:01.724-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:01.724-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:02.171-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.171-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:02.227-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.227-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:02.227-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 51 checks in a row. 2016-11-02T14:13:02.675-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.675-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:02.728-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:02.728-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:03.177-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.177-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:03.230-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.230-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:03.678-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.678-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:03.731-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:03.731-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:04.179-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.179-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:04.232-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.232-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:04.680-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.680-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:04.735-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:04.735-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:04.735-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 61 checks in a row. 2016-11-02T14:13:05.181-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.181-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:05.236-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.236-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:05.683-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.683-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:05.738-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:05.738-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:06.185-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:06.185-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:06.240-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:06.240-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:06.687-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:06.687-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:06.741-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:06.741-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:07.189-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:07.190-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:07.244-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:07.244-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:07.245-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 71 checks in a row. 2016-11-02T14:13:07.528-0400 I NETWORK [thread1] connection accepted from 10.0.1.3:59524 #9 (5 connections now open) 2016-11-02T14:13:07.528-0400 I NETWORK [conn9] received client metadata from 10.0.1.3:59524 conn9: { driver: { name: "MongoDB Internal Client", version: "3.4.0-rc2" }, os: { type: "Darwin", name: "Mac OS X", architecture: "x86_64", version: "16.1.0" } } 2016-11-02T14:13:07.691-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:07.692-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:07.745-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:07.745-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:08.194-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:08.194-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:08.246-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:08.247-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:08.698-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:08.698-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:08.751-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:08.751-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:09.203-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:09.203-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:09.252-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:09.252-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:09.705-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:09.705-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:09.753-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:09.753-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:09.753-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 81 checks in a row. 2016-11-02T14:13:10.209-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:10.209-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:10.209-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:13:10.257-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:10.257-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:10.257-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:13:20.125-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:20.126-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:13:40.212-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.212-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:40.257-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.257-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:40.714-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.714-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:40.762-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:40.762-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:41.214-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.214-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:41.266-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.266-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:41.716-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.716-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:41.716-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 91 checks in a row. 2016-11-02T14:13:41.769-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:41.770-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:42.222-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.222-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:42.274-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.274-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:42.728-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.728-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:42.776-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:42.776-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:43.231-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.231-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:43.281-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.281-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:43.733-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.733-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:43.783-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:43.783-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:44.235-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.235-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:44.235-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 101 checks in a row. 2016-11-02T14:13:44.288-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.288-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:44.741-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.741-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:44.792-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:44.792-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:45.243-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.243-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:45.296-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.296-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:45.746-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.746-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:45.797-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:45.798-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:46.247-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.247-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:46.300-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.300-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:46.752-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.752-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:46.752-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 111 checks in a row. 2016-11-02T14:13:46.802-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:46.803-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:47.257-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.257-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:47.307-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.307-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:47.761-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.762-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:47.808-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:47.808-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:48.265-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.265-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:48.310-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.310-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:48.771-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.771-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:48.814-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:48.814-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:49.276-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.276-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:49.276-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 121 checks in a row. 2016-11-02T14:13:49.316-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.316-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:49.777-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.778-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:49.820-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:49.820-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:50.126-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.126-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:13:50.281-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.281-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:50.324-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.324-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:50.787-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.787-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:50.825-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:50.825-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:51.288-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.288-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:51.328-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.328-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:51.328-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 131 checks in a row. 2016-11-02T14:13:51.790-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.790-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:51.833-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:51.833-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:52.295-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.295-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:52.336-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.336-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:52.796-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.796-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:52.837-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:52.837-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:53.297-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.297-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:53.340-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.340-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:53.800-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.800-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:53.844-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:53.844-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:53.844-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 141 checks in a row. 2016-11-02T14:13:54.301-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.301-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:54.345-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.345-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:54.804-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.804-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:54.846-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:54.846-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:55.309-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.309-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:55.347-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.347-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:55.812-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.812-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:55.848-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:55.848-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:56.313-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:56.313-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:56.349-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:56.349-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:56.349-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 151 checks in a row. 2016-11-02T14:13:56.817-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:56.817-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:56.851-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:56.852-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:57.321-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:57.321-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:57.352-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:57.352-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:57.824-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:57.824-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:57.856-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:57.856-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:58.330-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:58.330-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:58.359-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:58.359-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:58.832-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:58.832-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:58.864-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:58.864-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:58.864-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 161 checks in a row. 2016-11-02T14:13:59.335-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:59.335-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:59.365-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:59.365-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:13:59.837-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:59.837-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:13:59.866-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:13:59.866-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:00.339-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:00.339-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:00.339-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:14:00.371-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:00.371-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:00.371-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:14:20.129-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:20.129-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:14:30.341-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.341-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:30.373-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.373-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:30.847-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.847-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:30.847-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 171 checks in a row. 2016-11-02T14:14:30.877-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:30.878-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:31.351-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.351-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:31.383-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.383-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:31.852-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.852-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:31.885-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:31.885-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:32.354-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.354-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:32.391-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.391-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:32.856-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.856-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:32.892-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:32.892-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:33.360-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.360-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:33.360-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 181 checks in a row. 2016-11-02T14:14:33.393-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.394-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:33.861-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.861-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:33.895-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:33.895-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:34.362-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.362-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:34.398-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.398-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:34.866-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.866-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:34.900-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:34.900-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:35.370-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.370-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:35.405-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.406-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:35.874-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.874-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:35.874-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 191 checks in a row. 2016-11-02T14:14:35.908-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:35.908-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:36.379-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.379-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:36.414-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.414-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:36.884-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.884-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:36.915-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:36.915-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:37.386-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.386-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:37.416-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.416-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:37.887-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.887-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:37.919-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:37.920-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:38.389-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.389-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:38.389-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 201 checks in a row. 2016-11-02T14:14:38.422-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.422-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:38.890-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.890-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:38.926-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:38.926-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:39.391-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.391-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:39.428-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.428-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:39.892-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.892-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:39.933-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:39.933-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:40.393-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.394-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:40.435-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.436-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:40.895-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.895-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:40.895-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 211 checks in a row. 2016-11-02T14:14:40.939-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:40.939-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:41.399-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.399-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:41.445-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.445-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:41.905-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.905-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:41.945-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:41.945-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:42.410-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.410-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:42.446-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.446-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:42.911-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.912-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:42.949-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:42.949-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:43.416-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.416-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:43.416-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 221 checks in a row. 2016-11-02T14:14:43.451-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.451-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:43.921-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.921-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:43.952-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:43.952-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:44.424-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.424-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:44.455-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.455-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:44.929-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.930-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:44.959-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:44.959-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:45.433-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.433-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:45.463-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.463-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:45.936-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.936-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:45.936-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 231 checks in a row. 2016-11-02T14:14:45.967-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:45.967-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:46.439-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:46.439-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:46.469-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:46.469-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:46.941-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:46.941-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:46.970-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:46.970-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:47.442-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:47.442-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:47.475-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:47.475-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:47.947-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:47.947-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:47.980-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:47.981-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:48.453-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:48.453-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:48.453-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 241 checks in a row. 2016-11-02T14:14:48.482-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:48.482-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:48.958-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:48.958-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:48.983-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:48.983-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:49.460-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:49.460-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:49.485-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:49.485-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:49.963-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:49.963-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:49.988-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:49.988-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:50.129-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:50.129-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:14:50.468-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:50.468-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:14:50.468-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:14:50.494-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:14:50.494-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:14:50.494-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 251 checks in a row. 2016-11-02T14:14:50.494-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:15:20.131-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.132-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:15:20.468-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.469-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:20.493-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.493-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:20.970-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.970-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:20.996-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:20.996-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:21.473-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.473-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:21.497-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.497-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:21.974-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.974-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:21.999-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:21.999-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:22.477-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.477-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:22.477-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 261 checks in a row. 2016-11-02T14:15:22.503-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.503-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:22.979-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:22.979-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:23.007-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.007-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:23.481-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.481-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:23.509-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.509-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:23.983-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:23.983-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:24.009-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.010-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:24.488-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.488-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:24.515-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.515-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:24.992-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:24.992-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:24.992-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 271 checks in a row. 2016-11-02T14:15:25.017-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.017-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:25.496-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.496-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:25.522-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.522-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:25.997-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:25.997-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:26.024-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.026-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:26.499-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.499-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:26.527-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:26.527-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:27.000-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.000-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:27.028-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.028-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:27.500-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.500-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:27.500-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 281 checks in a row. 2016-11-02T14:15:27.533-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:27.534-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:28.001-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.001-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:28.035-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.035-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:28.505-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.505-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:28.537-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:28.538-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:29.009-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.009-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:29.038-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.038-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:29.510-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.510-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:29.542-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:29.543-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:30.013-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.013-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:30.013-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 291 checks in a row. 2016-11-02T14:15:30.045-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.045-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:30.517-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.518-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:30.550-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:30.550-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:31.021-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.021-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:31.053-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.053-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:31.526-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.526-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:31.554-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:31.555-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:32.029-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.029-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:32.059-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.059-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:32.535-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.535-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:32.535-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 301 checks in a row. 2016-11-02T14:15:32.560-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:32.560-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:33.041-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.041-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:33.063-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.063-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:33.543-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.543-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:33.568-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:33.568-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:34.048-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.048-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:34.071-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.071-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:34.549-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.549-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:34.572-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:34.572-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:35.052-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.052-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:35.052-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 311 checks in a row. 2016-11-02T14:15:35.078-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.078-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:35.558-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.558-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:35.580-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:35.580-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:36.062-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:36.062-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:36.083-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:36.083-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:36.566-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:36.566-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:36.584-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:36.584-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:37.068-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:37.068-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:37.086-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:37.086-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:37.574-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:37.574-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:37.574-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 321 checks in a row. 2016-11-02T14:15:37.586-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:37.586-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:38.075-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:38.075-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:38.087-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:38.087-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:38.576-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:38.576-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:38.588-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:38.588-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:39.077-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:39.077-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:39.092-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:39.092-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:39.583-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:39.583-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:39.595-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:39.595-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:40.085-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:40.085-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:40.085-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 331 checks in a row. 2016-11-02T14:15:40.097-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:40.097-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:40.588-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:40.588-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:15:40.588-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:15:40.599-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:40.599-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:15:40.599-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:15:50.133-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:15:50.133-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:16:10.588-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:10.588-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:10.599-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:10.599-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:11.092-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.092-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:11.102-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.102-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:11.596-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.596-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:11.604-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:11.604-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:11.604-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 341 checks in a row. 2016-11-02T14:16:12.099-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.099-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:12.109-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.109-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:12.601-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.601-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:12.615-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:12.615-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:13.103-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.103-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:13.117-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.117-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:13.608-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.608-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:13.617-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:13.617-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:14.109-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.109-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:14.121-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.121-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:14.121-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 351 checks in a row. 2016-11-02T14:16:14.612-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.612-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:14.623-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:14.623-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:15.114-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.114-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:15.124-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.124-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:15.618-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.618-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:15.627-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:15.627-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:16.120-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.120-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:16.131-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.131-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:16.625-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.625-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:16.633-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:16.633-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:16.633-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 361 checks in a row. 2016-11-02T14:16:17.126-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.126-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:17.137-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.137-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:17.627-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.627-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:17.641-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:17.641-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:18.129-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.129-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:18.141-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.141-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:18.631-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.631-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:18.643-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:18.643-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:19.136-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.136-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:19.147-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.147-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:19.147-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 371 checks in a row. 2016-11-02T14:16:19.637-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.637-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:19.648-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:19.648-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:20.136-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.136-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:16:20.138-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.138-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:20.151-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.151-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:20.640-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.640-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:20.652-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:20.652-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:21.143-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.143-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:21.157-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.157-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:21.647-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.648-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:21.648-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 381 checks in a row. 2016-11-02T14:16:21.658-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:21.658-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:22.153-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.153-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:22.160-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.160-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:22.658-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.658-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:22.662-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:22.662-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:23.160-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.160-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:23.166-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.166-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:23.661-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.661-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:23.671-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:23.671-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:23.865-0400 I - [conn5] end connection 10.0.1.3:59138 (5 connections now open) 2016-11-02T14:16:24.165-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.165-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:24.165-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 391 checks in a row. 2016-11-02T14:16:24.175-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.175-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:24.671-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.671-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:24.678-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:24.679-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:25.172-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.172-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:25.180-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.181-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:25.676-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.676-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:25.683-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:25.683-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:26.179-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.179-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:26.186-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.186-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:26.680-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.680-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:26.680-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 401 checks in a row. 2016-11-02T14:16:26.689-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:26.689-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:27.186-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:27.186-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:27.191-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:27.191-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:27.689-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:27.689-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:27.694-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:27.694-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:28.192-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:28.192-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:28.196-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:28.196-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:28.693-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:28.693-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:28.699-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:28.699-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:29.198-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:29.198-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:29.198-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 411 checks in a row. 2016-11-02T14:16:29.202-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:29.202-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:29.702-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:29.702-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:29.704-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:29.704-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:30.203-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:30.203-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:30.208-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:30.208-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:30.704-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:30.704-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:16:30.704-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:16:30.711-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:30.711-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:16:30.712-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:16:50.136-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:16:50.136-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:17:00.705-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.705-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:00.713-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:00.713-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:00.713-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 421 checks in a row. 2016-11-02T14:17:01.211-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.211-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:01.218-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.218-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:01.712-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.712-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:01.724-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:01.725-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:02.217-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.218-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:02.230-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.230-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:02.719-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.720-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:02.733-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:02.733-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:03.220-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.220-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:03.235-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.236-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:03.236-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 431 checks in a row. 2016-11-02T14:17:03.722-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.722-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:03.736-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:03.736-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:04.224-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.224-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:04.242-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.242-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:04.726-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.726-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:04.743-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:04.743-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:05.229-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.229-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:05.244-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.244-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:05.732-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.732-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:05.749-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:05.749-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:05.749-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 441 checks in a row. 2016-11-02T14:17:06.233-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.234-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:06.252-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.252-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:06.736-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.736-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:06.757-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:06.757-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:07.242-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.242-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:07.258-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.258-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:07.742-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.743-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:07.759-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:07.759-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:08.245-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.245-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:08.263-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.264-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:08.264-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 451 checks in a row. 2016-11-02T14:17:08.749-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.749-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:08.765-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:08.765-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:09.250-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.250-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:09.270-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.270-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:09.751-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.751-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:09.776-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:09.776-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:10.257-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.257-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:10.277-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.277-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:10.760-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.760-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:10.781-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:10.781-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:10.781-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 461 checks in a row. 2016-11-02T14:17:11.261-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.261-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:11.282-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.282-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:11.762-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.762-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:11.787-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:11.787-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:12.265-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.265-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:12.288-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.288-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:12.769-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.770-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:12.792-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:12.792-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:13.272-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.272-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:13.293-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.293-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:13.293-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 471 checks in a row. 2016-11-02T14:17:13.774-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.774-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:13.797-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:13.797-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:14.279-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.279-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:14.298-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.298-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:14.781-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.781-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:14.802-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:14.802-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:15.282-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.282-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:15.304-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.305-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:15.783-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.783-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:15.809-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:15.809-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:15.809-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 481 checks in a row. 2016-11-02T14:17:16.288-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.288-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:16.310-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.310-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:16.792-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.792-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:16.813-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:16.813-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:17.293-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:17.293-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:17.316-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:17.316-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:17.797-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:17.797-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:17.820-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:17.820-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:18.301-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:18.301-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:18.325-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:18.325-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:18.325-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 491 checks in a row. 2016-11-02T14:17:18.804-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:18.804-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:18.831-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:18.831-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:19.306-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:19.306-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:19.334-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:19.334-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:19.809-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:19.809-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:19.837-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:19.837-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:20.138-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:20.138-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:17:20.312-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:20.312-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:20.342-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:20.342-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:20.817-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:20.817-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:20.817-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 501 checks in a row. 2016-11-02T14:17:20.817-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:17:20.845-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:20.845-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:20.845-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:17:50.140-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.140-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:17:50.817-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.817-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:50.847-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:50.847-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:51.320-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.320-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:51.348-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.348-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:51.826-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.826-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:51.849-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:51.849-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:52.328-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.328-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:52.350-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.350-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:52.350-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 511 checks in a row. 2016-11-02T14:17:52.829-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.829-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:52.855-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:52.855-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:53.330-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.330-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:53.355-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.355-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:53.831-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.831-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:53.860-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:53.860-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:54.334-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.334-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:54.361-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.362-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:54.837-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.837-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:54.864-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:54.864-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:54.864-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 521 checks in a row. 2016-11-02T14:17:55.338-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.338-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:55.368-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.368-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:55.843-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.843-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:55.872-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:55.872-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:56.347-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.347-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:56.376-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.376-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:56.849-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.849-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:56.880-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:56.880-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:57.350-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.350-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:57.381-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.381-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:57.381-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 531 checks in a row. 2016-11-02T14:17:57.856-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.856-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:57.886-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:57.886-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:58.361-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.361-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:58.389-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.389-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:58.864-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.864-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:58.891-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:58.891-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:59.365-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.365-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:59.397-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.397-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:59.869-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.869-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:17:59.900-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:17:59.900-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:17:59.900-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 541 checks in a row. 2016-11-02T14:18:00.370-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.370-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:00.401-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.401-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:00.875-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.875-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:00.903-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:00.904-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:01.378-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.378-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:01.406-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.406-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:01.880-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.880-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:01.911-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:01.911-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:02.384-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.384-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:02.412-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.412-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:02.412-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 551 checks in a row. 2016-11-02T14:18:02.888-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.888-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:02.917-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:02.917-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:03.391-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.391-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:03.420-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.420-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:03.894-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.894-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:03.921-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:03.921-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:04.399-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.399-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:04.422-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.422-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:04.904-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.904-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:04.925-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:04.925-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:04.925-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 561 checks in a row. 2016-11-02T14:18:05.408-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.408-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:05.426-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.426-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:05.911-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.911-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:05.927-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:05.927-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:06.412-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:06.412-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:06.428-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:06.428-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:06.916-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:06.916-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:06.932-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:06.932-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:07.418-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:07.418-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:07.434-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:07.434-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:07.434-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 571 checks in a row. 2016-11-02T14:18:07.921-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:07.921-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:07.938-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:07.939-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:08.424-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:08.425-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:08.439-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:08.439-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:08.926-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:08.926-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:08.942-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:08.942-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:09.431-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:09.431-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:09.444-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:09.444-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:09.932-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:09.932-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:09.950-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:09.950-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:09.950-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 581 checks in a row. 2016-11-02T14:18:10.437-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:10.438-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:10.453-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:10.453-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:10.941-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:10.941-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:10.942-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:18:10.953-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:10.953-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:10.953-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:18:20.141-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:20.142-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:18:40.945-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:40.945-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:40.956-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:40.956-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:41.446-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.446-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:41.459-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.459-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:41.947-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.948-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:41.948-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 591 checks in a row. 2016-11-02T14:18:41.961-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:41.961-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:42.451-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.451-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:42.465-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.465-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:42.954-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.954-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:42.969-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:42.969-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:43.455-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.455-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:43.470-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.470-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:43.958-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.958-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:43.971-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:43.971-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:44.463-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.463-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:44.463-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 601 checks in a row. 2016-11-02T14:18:44.472-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.472-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:44.967-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.967-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:44.973-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:44.973-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:45.471-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.471-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:45.478-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.478-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:45.973-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.973-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:45.982-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:45.982-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:46.479-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.479-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:46.486-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.486-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:46.982-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.982-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:46.982-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 611 checks in a row. 2016-11-02T14:18:46.987-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:46.987-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:47.487-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.488-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:47.491-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.491-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:47.992-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:47.992-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:48.495-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.495-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:48.996-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:48.996-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:49.501-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:49.501-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:50.004-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.004-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:50.142-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.142-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:18:50.508-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:50.508-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:50.508-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 621 checks in a row. 2016-11-02T14:18:51.010-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.010-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:51.511-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:51.511-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:52.012-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.012-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:52.518-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:52.518-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:53.021-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.021-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:53.525-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:53.525-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:54.026-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.026-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:54.531-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:54.531-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:55.034-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.034-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:55.535-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:55.535-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:55.535-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 631 checks in a row. 2016-11-02T14:18:56.040-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:56.040-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:56.541-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:56.541-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:57.045-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:57.045-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:57.546-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:57.546-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:18:58.049-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:58.049-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:58.549-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:58.549-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:59.051-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:59.051-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:18:59.556-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:18:59.556-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:00.060-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:00.060-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:00.563-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:00.563-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:00.563-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 641 checks in a row. 2016-11-02T14:19:01.064-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:01.064-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:01.064-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:19:01.064-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:19:20.143-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:20.143-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:19:31.064-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:31.065-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:31.565-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:31.565-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:32.067-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:32.067-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:32.571-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:32.571-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:33.077-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:33.077-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:33.578-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:33.578-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:34.080-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:34.080-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:34.581-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:34.581-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:34.581-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 651 checks in a row. 2016-11-02T14:19:35.087-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:35.087-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:35.591-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:35.591-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:36.092-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:36.092-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:36.597-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:36.597-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:37.099-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:37.099-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:37.603-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:37.603-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:38.106-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:38.106-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:38.610-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:38.610-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:39.114-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:39.114-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:39.617-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:39.617-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:39.617-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 661 checks in a row. 2016-11-02T14:19:40.121-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:40.121-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:40.622-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:40.622-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:41.124-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:41.124-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:41.625-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:41.625-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:42.130-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:42.130-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:42.632-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:42.633-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:43.137-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:43.137-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:43.639-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:43.639-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:44.141-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:44.141-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:44.643-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:44.643-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:44.643-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 671 checks in a row. 2016-11-02T14:19:45.148-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:45.148-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:45.653-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:45.653-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:46.154-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:46.155-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:46.660-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:46.660-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:47.162-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:47.163-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:47.663-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:47.663-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:48.169-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:48.169-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:19:48.672-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:48.672-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:49.176-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:49.176-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:49.678-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:49.678-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:49.678-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 681 checks in a row. 2016-11-02T14:19:50.143-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:50.143-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:19:50.180-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:50.180-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:50.684-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:50.684-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:51.186-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:19:51.186-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:19:51.186-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: FailedToSatisfyReadPreference: could not find host matching read preference { mode: "primary" } for set csrs 2016-11-02T14:19:51.186-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 133 could not get updated shard list from config server due to could not find host matching read preference { mode: "nearest" } for set csrs; will retry after 30s 2016-11-02T14:20:20.144-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:20.144-0400 W NETWORK [ReplicaSetMonitor-TaskExecutor-0] No primary detected for set csrs 2016-11-02T14:20:21.186-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.186-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:21.690-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:21.691-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:22.194-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.194-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:22.698-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:22.698-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:23.202-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.202-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:23.202-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 691 checks in a row. 2016-11-02T14:20:23.706-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:23.706-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:24.208-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.209-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:24.710-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:24.710-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:25.213-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.213-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:25.714-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:25.714-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:26.218-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.218-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:26.719-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:26.719-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:27.219-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.219-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:27.722-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:27.722-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:28.225-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.226-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:28.226-0400 I NETWORK [replSetDistLockPinger] All nodes for set csrs are down. This has happened for 701 checks in a row. 2016-11-02T14:20:28.730-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:28.730-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:29.235-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.235-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:29.736-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:29.736-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:30.237-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:30.237-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:30.740-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:30.740-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:31.241-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.241-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:31.742-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:31.742-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:32.246-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.246-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:32.749-0400 W NETWORK [replSetDistLockPinger] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:32.749-0400 W NETWORK [replSetDistLockPinger] No primary detected for set csrs 2016-11-02T14:20:33.254-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:33.254-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:33.254-0400 I NETWORK [shard registry reload] All nodes for set csrs are down. This has happened for 711 checks in a row. 2016-11-02T14:20:33.755-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:33.755-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:33.824-0400 I - [conn7] end connection 10.0.1.3:59147 (4 connections now open) 2016-11-02T14:20:34.259-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:34.259-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:34.763-0400 W NETWORK [shard registry reload] Failed to connect to 10.0.1.3:27002, reason: Connection refused 2016-11-02T14:20:34.763-0400 W NETWORK [shard registry reload] No primary detected for set csrs 2016-11-02T14:20:34.851-0400 I CONTROL [signalProcessingThread] got signal 2 (Interrupt: 2), will terminate after current cmd ends 2016-11-02T14:20:34.851-0400 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets... 2016-11-02T14:20:34.851-0400 I NETWORK [signalProcessingThread] closing listening socket: 7 2016-11-02T14:20:34.851-0400 I NETWORK [signalProcessingThread] closing listening socket: 8 2016-11-02T14:20:34.851-0400 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-27001.sock 2016-11-02T14:20:34.851-0400 I NETWORK [signalProcessingThread] shutdown: going to flush diaglog... 2016-11-02T14:20:34.851-0400 I REPL [signalProcessingThread] shutting down replication subsystems 2016-11-02T14:20:34.851-0400 I REPL [signalProcessingThread] Stopping replication reporter thread 2016-11-02T14:20:34.851-0400 I REPL [signalProcessingThread] Stopping replication fetcher thread 2016-11-02T14:20:34.851-0400 I REPL [signalProcessingThread] Stopping replication applier thread 2016-11-02T14:20:35.178-0400 I REPL [signalProcessingThread] Stopping replication storage threads 2016-11-02T14:20:35.266-0400 W SHARDING [replSetDistLockPinger] pinging failed for distributed lock pinger :: caused by :: InterruptedAtShutdown: interrupted at shutdown 2016-11-02T14:20:35.266-0400 I SHARDING [shard registry reload] Periodic reload of shard registry failed :: caused by :: 11600 could not get updated shard list from config server due to interrupted at shutdown; will retry after 30s 2016-11-02T14:20:35.267-0400 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture 2016-11-02T14:20:35.269-0400 W SHARDING [signalProcessingThread] error encountered while cleaning up distributed ping entry for liquid.local:27001:1478110280:3025486684967422951 :: caused by :: InterruptedAtShutdown: interrupted at shutdown 2016-11-02T14:20:35.269-0400 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down 2016-11-02T14:20:35.410-0400 I STORAGE [signalProcessingThread] shutdown: removing fs lock... 2016-11-02T14:20:35.411-0400 I CONTROL [signalProcessingThread] now exiting 2016-11-02T14:20:35.411-0400 I CONTROL [signalProcessingThread] shutting down with code:0