We had an issue today where, in a 3 member replicaset, when one of the secondaries becomes angry and starts erroring out on simple connectivity, that our entire site may go down in a 503 scenario.

SERVER-25663 is the related issue. We are unable to reproduce, because the scenario in which the secondary got into its mess is uncertain.

We are sure, however, that there must be some unhandled exception coming back in the client connectivity, causing pure failure of the site. It may be an unexpected network error.

Simply shutting down the angry secondary immediately fixed the issue. When the angry secondary came back up, it syncd and all remained well.