When a shard primary relinquishes, it closes all incoming — and outgoing — connections. This is normal and necessary. However, if it later becomes primary again, it will incorrectly try to reuse the (now closed) outgoing sockets to the configsvrs and other shards members (ReplicaSetMonitorWatcher).

Since these fds have been closed and are no longer valid, this causes a profusion of "Bad file descriptor" (errno = EBADF) messages in the logfile. However, the connections are not automatically re-established, causing subsequent chunk migrations to fail (and probably other operations that require the shards to write to the configsvrs).

The actual impact depends on whether the FROM or TO shard has "bounced" (step-down/step-up).

• FROM shard bounce => next 4 migrations fail
• TO shard bounce => next 3 migrations fail
• FROM and TO shard bounce => next 8 migrations fail

Initially the failures are early in the migration process, but subsequent migrations fail later in the process — notably, after documents have been transferred (causing orphaned documents). In some of these failures, SERVER-17066 means that the resulting orphans cannot be cleaned by cleanupOrphaned.

Attached are a jstest reproducer and wrapper script suitable for "git bisect run".

This only affects 2.6; it has been incidentally fixed in 3.0. Using git bisect shows that commit fbbb0d2a1d845728cd714272199a652573e2f27d (SERVER-15593) fixed the issue. However, that ticket is different and the bulk of the commit is completely unrelated.

I have confirmed that the following hunk alone is sufficient to fix the problem:

diff --git a/src/mongo/util/net/sock.cpp b/src/mongo/util/net/sock.cpp
index 8e9517f..e649a43 100644
--- a/src/mongo/util/net/sock.cpp
+++ b/src/mongo/util/net/sock.cpp
@@ -824,6 +824,12 @@ namespace mongo {
     // isStillConnected() polls the socket at max every Socket::errorPollIntervalSecs to determine
     // if any disconnection-type events have happened on the socket.
     bool Socket::isStillConnected() {
+        if (_fd == -1) {
+            // According to the man page, poll will respond with POLLVNAL for invalid or
+            // unopened descriptors, but it doesn't seem to be properly implemented in
+            // some platforms - it can return 0 events and 0 for revent. Hence this workaround.
+            return false;
+        }

         if ( errorPollIntervalSecs < 0 ) return true;
         if ( ! isPollSupported() ) return true; // nothing we can do

Given that this is a very simple fix for a logic bug of moderately high impact, can this please be backported to the v2.6 branch?