Of the 3 node CSRS, one node is consistently having connection refused problems as is seen in it's own log as well as other config servers and all mongos logs:

2016-10-19T09:01:10.957-0400 W NETWORK  [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused
2016-10-19T09:01:21.945-0400 W NETWORK  [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused
2016-10-19T09:01:32.933-0400 W NETWORK  [ReplicaSetMonitorWatcher] Failed to connect to 10.170.7.16:29102, reason: errno:111 Connection refused

This is the 3rd node of the CSRS and has dbpath pointing to a netapp volume. Note that the limits are bumped up pretty high and there is minimal load on the cluster so it should not have anything to do with lack of available sockets/file descriptors to open.

The sharded cluster is running MongoDb 3.2.5 with WiredTiger.