[SERVER-20740] External exception not handled well by SASL/LDAP implementation Created: 02/Oct/15  Updated: 07/Dec/16  Resolved: 03/Jun/16

Status: Closed
Project: Core Server
Component/s: Networking, Security
Affects Version/s: 2.6.9
Fix Version/s: 3.3.8

Type: Bug Priority: Major - P3
Reporter: Steven Hand Assignee: Spencer Jackson
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-21576 Improve test coverage for SASL authen... Closed
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

1) Step sharded cluster
2) Make config servers unresponsive
3) Attempt authorization through mongos using SASL
4) Watch for MongoS crash

Sprint: Security B 10/30/15, Security C 11/20/15, Security D (12/11/15), Security E (01/01/16), Security F (01/29/16), Security 10 (02/19/16), Security 11 (03/11/16), Security 12 (04/01/16), Security 13 (04/22/16), Security 15 (06/03/16)
Participants:

 Description   

The Mongos authorization code does not handle exceptions raised by the Cyrus SASL implementation and crashes.



 Comments   
Comment by Githook User [ 03/Jun/16 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-20740: Check exceptions in SASL callbacks
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/fcdf816d32f1c4a218a8c7e39859c689dc68be5f

Comment by Githook User [ 03/Jun/16 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-20740: Check exceptions in SASL callbacks
Branch: master
https://github.com/mongodb/mongo/commit/94b726920c165c88b32e31d191198db89a28686e

Comment by Steven Hand [ 02/Oct/15 ]

The versions of OS and supporting libraries in the environment experiencing the issue:

cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.7 (Santiago)
 
rpm -qa | grep -i sasl
cyrus-sasl-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-gssapi-2.1.23-15.el6_6.2.x86_64
cyrus-sasl-devel-2.1.23-15.el6_6.2.x86_64
 
rpm -qa | grep -i krb5
krb5-libs-1.10.3-42.el6.x86_64
krb5-workstation-1.10.3-42.el6.x86_64
 
rpm -qa | grep -i glibc
glibc-2.12-1.166.el6_7.1.i686
glibc-headers-2.12-1.166.el6_7.1.x86_64
glibc-2.12-1.166.el6_7.1.x86_64
glibc-devel-2.12-1.166.el6_7.1.x86_64
glibc-common-2.12-1.166.el6_7.1.x86_64

Comment by Steven Hand [ 02/Oct/15 ]

The operative theory is that an exception is being thrown from within MongoS's authorization code when attempting to acquire a password from the configure servers on the behalf of Cyrus SASL. A guess is that while creating ScopedDbConnection to the config server, the constructor timed out and threw an exception which propagated back to our call back, and tried to pass into C code.

A possible solution would be wrapping the authorization logic in the Enterprise Module with a try/catch block to consume any and all exception raised by the internal authorization code before they propagate back to C (while still within the C++ code).

Comment by Steven Hand [ 02/Oct/15 ]

This issue was experience with a configuration of three {{mongos} servers and a config server.

The stack trace produced was:

2015-09-29T23:49:12.108-0400 [conn61635] SEVERE: terminate() called, printing stack (if implemented for platform):
0xc6c206 0xc6bb00 0x7fd7425debd6 0x7fd7425dec03 0x7fd7425dec16 0x7fd7425de2a3 0x920f65 0x7fd7434e4f8a 0x7fd7434e67a9 0x7fd739b2abf2 0x7fd7434ef8b1 0x9237b8 0x9265ae 0x9286fb 0xb510a3 0xac0c59 0xb6d442 0xb4fc6a 0x728999 0xc28d39
/usr/bin/mongos(_ZN5mongo15printStackTraceERSo+0x26) [0xc6c206]
/usr/bin/mongos() [0xc6bb00]
/usr/lib64/libstdc++.so.6(+0xbcbd6) [0x7fd7425debd6]
/usr/lib64/libstdc++.so.6(+0xbcc03) [0x7fd7425dec03]
/usr/lib64/libstdc++.so.6(+0xbcc16) [0x7fd7425dec16]
/usr/lib64/libstdc++.so.6(__cxa_call_unexpected+0x43) [0x7fd7425de2a3]
/usr/bin/mongos() [0x920f65]
/usr/lib64/libsasl2.so.2(_sasl_auxprop_lookup+0x15a) [0x7fd7434e4f8a]
/usr/lib64/libsasl2.so.2(_sasl_canon_user+0x389) [0x7fd7434e67a9]
/usr/lib64/sasl2/libgssapiv2.so(+0x3bf2) [0x7fd739b2abf2]
/usr/lib64/libsasl2.so.2(sasl_server_step+0xb1) [0x7fd7434ef8b1]
/usr/bin/mongos(_ZN5mongo25SaslAuthenticationSession4stepERKNS_10StringDataEPSs+0xf8) [0x9237b8]
/usr/bin/mongos() [0x9265ae]
/usr/bin/mongos() [0x9286fb]
/usr/bin/mongos(_ZN5mongo7Command22execCommandClientBasicEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderEb+0x3d3) [0xb510a3]
/usr/bin/mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x349) [0xac0c59]
/usr/bin/mongos(_ZN5mongo8Strategy15clientCommandOpERNS_7RequestE+0x3d2) [0xb6d442]
/usr/bin/mongos(_ZN5mongo7Request7processEi+0x7fa) [0xb4fc6a]
/usr/bin/mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortEPNS_9LastErrorE+0x69) [0x728999]
/usr/bin/mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x5c9) [0xc28d39]

Generated at Thu Feb 08 03:55:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.