Major - P3 Affects 2.2.x only.
When 1 config server is down, about ~1/16 authenticate commands (per thread) will incorrectly fail with the error message "exception: SyncClusterConnection::insert prepare failed: 9001 socket exception".
The authenticate command in mongos uses a helper that has an OCCASIONALLY call to create a
{user:1}index on system.users. If a config server is down when this call is made, the index creation will fail (as expected), but the socket exception goes uncaught and causes failure of the authenticate command.
OCCASIONALLY call in CmdAuthenticate::getUserObj() (src/mongo/s/security.cpp):
47 OCCASIONALLY conn->get()->ensureIndex(systemUsers, userPattern, false, "user_1");
Stack trace:
Thu Jun 13 16:07:09 [conn127] warning: DBException thrown :: caused by :: 9001 socket exception [CONNECT_ERROR] for localhost:27019
0x814d51 0x7dd559 0x50f41b 0x51f8df 0x52af95 0x521683 0x522032 0x5276de 0x527c91 0x5201c0 0x59c483 0x59c993 0x59ce90 0x5183ff 0x764b0b 0x691a57 0x701654 0x796167 0x787cf3 0x760cbb
mongos(_ZN5mongo15printStackTraceERSo+0x21) [0x814d51]
mongos(_ZN5mongo11DBException13traceIfNeededERKS0_+0x129) [0x7dd559]
mongos(_ZN5mongo15SocketExceptionC1ENS0_4TypeESsiSs+0x11b) [0x50f41b]
mongos(_ZN5mongo18DBClientConnection16_checkConnectionEv+0x4af) [0x51f8df]
mongos(_ZN5mongo18DBClientConnection5queryERKSsNS_5QueryEiiPKNS_7BSONObjEii+0x135) [0x52af95]
mongos(_ZN5mongo17DBClientInterface5findNERSt6vectorINS_7BSONObjESaIS2_EERKSsNS_5QueryEiiPKS2_i+0xa3) [0x521683]
mongos(_ZN5mongo17DBClientInterface7findOneERKSsRKNS_5QueryEPKNS_7BSONObjEi+0x72) [0x522032]
mongos(_ZN5mongo20DBClientWithCommands10runCommandERKSsRKNS_7BSONObjERS3_iPKNS_19AuthenticationTableE+0x1ae) [0x5276de]
mongos(_ZN5mongo18DBClientConnection10runCommandERKSsRKNS_7BSONObjERS3_iPKNS_19AuthenticationTableE+0x11) [0x527c91]
mongos(_ZN5mongo20DBClientWithCommands13simpleCommandERKSsPNS_7BSONObjES2_+0x130) [0x5201c0]
mongos(_ZN5mongo21SyncClusterConnection5fsyncERSs+0x143) [0x59c483]
mongos(_ZN5mongo21SyncClusterConnection7prepareERSs+0xa3) [0x59c993]
mongos(_ZN5mongo21SyncClusterConnection6insertERKSsNS_7BSONObjEi+0x70) [0x59ce90]
mongos(_ZN5mongo20DBClientWithCommands11ensureIndexERKSsNS_7BSONObjEbS2_bbi+0x6af) [0x5183ff]
mongos(_ZN5mongo15CmdAuthenticate10getUserObjERKSsS2_RNS_7BSONObjERSs+0x29b) [0x764b0b]
mongos(_ZN5mongo15CmdAuthenticate3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x827) [0x691a57]
mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x764) [0x701654]
mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x467) [0x796167]
mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xaa3) [0x787cf3]
mongos(_ZN5mongo7Request7processEi+0x1fb) [0x760cbb]
Thu Jun 13 16:07:09 [conn127] warning: DBException thrown :: caused by :: 8003 SyncClusterConnection::insert prepare failed: 9001 socket exception [6] server [localhost:27019] localhost:27019:{}
0x814d51 0x7dd559 0x59cf25 0x5183ff 0x764b0b 0x691a57 0x701654 0x796167 0x787cf3 0x760cbb 0x500dd1 0x802f35 0x7f0cad36ee9a 0x7f0cac681ccd
mongos(_ZN5mongo15printStackTraceERSo+0x21) [0x814d51]
mongos(_ZN5mongo11DBException13traceIfNeededERKS0_+0x129) [0x7dd559]
mongos(_ZN5mongo21SyncClusterConnection6insertERKSsNS_7BSONObjEi+0x105) [0x59cf25]
mongos(_ZN5mongo20DBClientWithCommands11ensureIndexERKSsNS_7BSONObjEbS2_bbi+0x6af) [0x5183ff]
mongos(_ZN5mongo15CmdAuthenticate10getUserObjERKSsS2_RNS_7BSONObjERSs+0x29b) [0x764b0b]
mongos(_ZN5mongo15CmdAuthenticate3runERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderEb+0x827) [0x691a57]
mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x764) [0x701654]
mongos(_ZN5mongo14SingleStrategy7queryOpERNS_7RequestE+0x467) [0x796167]
mongos(_ZN5mongo13ShardStrategy7queryOpERNS_7RequestE+0xaa3) [0x787cf3]
mongos(_ZN5mongo7Request7processEi+0x1fb) [0x760cbb]
mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x71) [0x500dd1]
mongos(_ZN5mongo3pms9threadRunEPNS_13MessagingPortE+0x415) [0x802f35]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f0cad36ee9a]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f0cac681ccd]