Many thousands of these in master mongodb.log:

Tue Apr 12 07:05:23 [ReplSetHealthPollTask] MessagingPort recv() errno:104 Connection reset by peer 10.120.14.79:27018
Tue Apr 12 07:05:23 [ReplSetHealthPollTask] SocketException: remote: error: 9001 socket exception [1]
Tue Apr 12 07:05:23 [ReplSetHealthPollTask] DBClientCursor::init call() failed
Tue Apr 12 07:05:23 [ReplSetHealthPollTask] replSet info mongodb05.example.com:27018 is down (or slow to respond): DBClientBase::findOne: transport error: 10.120.14.79:27018 query:

Tue Apr 12 07:05:25 [ReplSetHealthPollTask] ERROR: MessagingPort::call() wrong id got:7c2a3e4f expect:7c2d5a40
toSend op: 2004
response msgid:2117972757
response len: 129
response op: 1
farEnd: 10.120.14.79:27018
Tue Apr 12 07:05:25 [ReplSetHealthPollTask] Assertion failure false util/message.cpp 512
0x54e8de 0x55f971 0x5593ae 0x55a5c4 0x586a94 0x598e30 0x57a69a 0x58a791 0x58633c 0x58994c 0x6e1bcb 0x6e5871 0x5650f3 0x542deb 0x5446a4 0x8b6a40 0x7f0a30bee9ca 0x7f0a3019d70d
/usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54e8de]
/usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x111) [0x55f971]
/usr/bin/mongod(ZN5mongo13MessagingPort4recvERKNS_7MessageERS1+0x32e) [0x5593ae]
/usr/bin/mongod(ZN5mongo13MessagingPort4callERNS_7MessageES2+0x34) [0x55a5c4]
/usr/bin/mongod(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPSs+0x34) [0x586a94]
/usr/bin/mongod(_ZN5mongo14DBClientCursor4initEv+0x150) [0x598e30]
/usr/bin/mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x32a) [0x57a69a]
/usr/bin/mongod(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa1) [0x58a791]
/usr/bin/mongod(_ZN5mongo17DBClientInterface7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi+0x7c) [0x58633c]
/usr/bin/mongod(_ZN5mongo20DBClientWithCommands10runCommandERKSsRKNS_7BSONObjERS3_i+0x8c) [0x58994c]
/usr/bin/mongod(_ZN5mongo16requestHeartbeatESsSsSsRNS_7BSONObjEiRib+0x6db) [0x6e1bcb]
/usr/bin/mongod(_ZN5mongo21ReplSetHealthPollTask6doWorkEv+0x221) [0x6e5871]
/usr/bin/mongod(_ZN5mongo4task4Task3runEv+0x33) [0x5650f3]
/usr/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x12b) [0x542deb]
/usr/bin/mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74) [0x5446a4]
/usr/bin/mongod(thread_proxy+0x80) [0x8b6a40]
/lib/libpthread.so.0(+0x69ca) [0x7f0a30bee9ca]
/lib/libc.so.6(clone+0x6d) [0x7f0a3019d70d]

Many thousands of these in arbiter mongodb.log:

Tue Apr 12 07:05:27 [ReplSetHealthPollTask] MessagingPort recv() errno:104 Connection reset by peer 10.120.14.79:27018
Tue Apr 12 07:05:27 [ReplSetHealthPollTask] SocketException: remote: error: 9001 socket exception [1]
Tue Apr 12 07:05:27 [ReplSetHealthPollTask] DBClientCursor::init call() failed
Tue Apr 12 07:05:27 [ReplSetHealthPollTask] replSet info mongodb05.example.com:27018 is down (or slow to respond): DBClientBase::findOne: transport error: 10.120.14.79:27018 query:

Tue Apr 12 07:05:29 [ReplSetHealthPollTask] ERROR: MessagingPort::call() wrong id got:3897648b expect:38976a03
toSend op: 2004
response msgid:2117972758
response len: 129
response op: 1
farEnd: 10.120.14.79:27018
Tue Apr 12 07:05:29 [ReplSetHealthPollTask] Assertion failure false util/message.cpp 512
0x54e8de 0x55f971 0x5593ae 0x55a5c4 0x586a94 0x598e30 0x57a69a 0x58a791 0x58633c 0x58994c 0x6e1bcb 0x6e5871 0x5650f3 0x542deb 0x5446a4 0x8b6a40 0x7fe632d719ca 0x7fe63232070d
/usr/bin/mongod(_ZN5mongo12sayDbContextEPKc+0xae) [0x54e8de]
/usr/bin/mongod(_ZN5mongo8assertedEPKcS1_j+0x111) [0x55f971]
/usr/bin/mongod(ZN5mongo13MessagingPort4recvERKNS_7MessageERS1+0x32e) [0x5593ae]
/usr/bin/mongod(ZN5mongo13MessagingPort4callERNS_7MessageES2+0x34) [0x55a5c4]
/usr/bin/mongod(_ZN5mongo18DBClientConnection4callERNS_7MessageES2_bPSs+0x34) [0x586a94]
/usr/bin/mongod(_ZN5mongo14DBClientCursor4initEv+0x150) [0x598e30]
/usr/bin/mongod(_ZN5mongo12DBClientBase5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x32a) [0x57a69a]
/usr/bin/mongod(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0xa1) [0x58a791]
/usr/bin/mongod(_ZN5mongo17DBClientInterface7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi+0x7c) [0x58633c]
/usr/bin/mongod(_ZN5mongo20DBClientWithCommands10runCommandERKSsRKNS_7BSONObjERS3_i+0x8c) [0x58994c]
/usr/bin/mongod(_ZN5mongo16requestHeartbeatESsSsSsRNS_7BSONObjEiRib+0x6db) [0x6e1bcb]
/usr/bin/mongod(_ZN5mongo21ReplSetHealthPollTask6doWorkEv+0x221) [0x6e5871]
/usr/bin/mongod(_ZN5mongo4task4Task3runEv+0x33) [0x5650f3]
/usr/bin/mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0x12b) [0x542deb]
/usr/bin/mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x74) [0x5446a4]
/usr/bin/mongod(thread_proxy+0x80) [0x8b6a40]
/lib/libpthread.so.0(+0x69ca) [0x7fe632d719ca]
/lib/libc.so.6(clone+0x6d) [0x7fe63232070d]

The only way I've found to resolve this is to restart the servers with these messages.