When a shard primary relinquishes, it closes all incoming — and outgoing — connections. This is normal and necessary. However, if it later becomes primary again, it will incorrectly try to reuse the (now closed) outgoing sockets to the configsvrs and other shards members (ReplicaSetMonitorWatcher).
Since these fds have been closed and are no longer valid, this causes a profusion of "Bad file descriptor" (errno = EBADF) messages in the logfile. However, the connections are not automatically re-established, causing subsequent chunk migrations to fail (and probably other operations that require the shards to write to the configsvrs).
The actual impact depends on whether the FROM or TO shard has "bounced" (step-down/step-up).
- FROM shard bounce => next 4 migrations fail
- TO shard bounce => next 3 migrations fail
- FROM and TO shard bounce => next 8 migrations fail
Initially the failures are early in the migration process, but subsequent migrations fail later in the process — notably, after documents have been transferred (causing orphaned documents). In some of these failures, SERVER-17066 means that the resulting orphans cannot be cleaned by cleanupOrphaned.
Attached are a jstest reproducer and wrapper script suitable for "git bisect run".
This only affects 2.6; it has been incidentally fixed in 3.0. Using git bisect shows that commit fbbb0d2a1d845728cd714272199a652573e2f27d (
SERVER-15593) fixed the issue. However, that ticket is different and the bulk of the commit is completely unrelated.
I have confirmed that the following hunk alone is sufficient to fix the problem:
Given that this is a very simple fix for a logic bug of moderately high impact, can this please be backported to the v2.6 branch?