A replica set member became unresponsive – connections could be established, but queries failed due to bad disk. It is very possible that rs.isMaster() continued to succeed. The member was later taken out of the replica set and brought down. The application reads from the replica set with a ReadPreference.SECONDARY_PREFERRED.

When the member was in the replica set but unreachable, the application behaved as expected – no reads were routed to the member and no exceptions in the logs.

After the member was removed from the replica set, reads were routed to the missing member and resulted in NetworkExceptions (triggered by SocketTimeoutExceptions). This continued to happen for over 12 hours until the application was restarted.

This only happened for application server there were running at the time of the re.reconfig(). Applications that were started shortly after the reconfig() did not exhibit the above behavior.

Write operations were unaffected across all applications.