Initial connects and later refreshes have a timeout associated with them in ASIO that isn't linked to any user generated timeout. These timeouts, when they trigger, are registered as general failures however. And general failures cause us to dump all connections from the pool (propagating that error to all consumers currently waiting for a connection).
That scheme is sound for actual io errors (because a failure to rpc on one connection almost certainly means something is badly wrong with all other open connections), but causes us to fail early and often when applied to timeouts.
The fix is to treat timeouts on connect and refresh lightly (start connecting a new connection on timeout) and allow the general request timeouts to handle timing out user requests