[SERVER-23394] AuthorizationManager may deadlock while building role graph if profiling is enabled Created: 29/Mar/16  Updated: 22/Nov/16  Resolved: 20/Apr/16

Status: Closed
Project: Core Server
Component/s: Security
Affects Version/s: 3.2.4, 3.3.3
Fix Version/s: 3.2.6, 3.3.5

Type: Bug Priority: Major - P3
Reporter: Benety Goh Assignee: Spencer Jackson
Resolution: Done Votes: 0
Labels: code-and-test
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-23461 Secondary stay STARTUP2 state because... Closed
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Sprint: Security 12 (04/01/16), Security 13 (04/22/16)
Participants:

 Description   

If profiling is enabled during authorization manager initialization, AuthzManagerLogOpHandler::commit() may deadlock waiting on the same mutex held by AuthzManagerExternalStateLocal::_initializeRoleGraph().

(lldb) thread backtrace 
* thread #24: tid = 0xefaaf, 0x00007fff92c40166 libsystem_kernel.dylib`__psynch_mutexwait + 10
  * frame #0: 0x00007fff92c40166 libsystem_kernel.dylib`__psynch_mutexwait + 10
    frame #1: 0x00007fff96669696 libsystem_pthread.dylib`_pthread_mutex_lock + 480
    frame #2: 0x00007fff94b62b6f libc++.1.dylib`std::__1::mutex::lock() + 9
    frame #3: 0x000000010e31bbf8 mongod`mongo::AuthzManagerExternalStateLocal::AuthzManagerLogOpHandler::commit() [inlined] std::__1::lock_guard<std::__1::mutex>::lock_guard(this=0x0000000117ba74f0, __m=0x00007f964044eeb0) + 120 at __mutex_base:83
    frame #4: 0x000000010e31bbdc mongod`mongo::AuthzManagerExternalStateLocal::AuthzManagerLogOpHandler::commit() [inlined] std::__1::lock_guard<std::__1::mutex>::lock_guard(this=0x0000000117ba74f0, __m=0x00007f964044eeb0) + 28 at __mutex_base:83
    frame #5: 0x000000010e31bbc0 mongod`mongo::AuthzManagerExternalStateLocal::AuthzManagerLogOpHandler::commit(this=0x00007f964070d700) + 64 at authz_manager_external_state_local.cpp:383
    frame #6: 0x000000010f1a0a05 mongod`mongo::WiredTigerRecoveryUnit::_commit(this=0x00007f9640502660) + 341 at wiredtiger_recovery_unit.cpp:86
    frame #7: 0x000000010f1a1039 mongod`mongo::WiredTigerRecoveryUnit::commitUnitOfWork(this=0x00007f9640502660) + 89 at wiredtiger_recovery_unit.cpp:127
    frame #8: 0x000000010e107587 mongod`mongo::WriteUnitOfWork::commit(this=0x0000000117ba7838) + 183 at operation_context.h:271
    frame #9: 0x000000010e862207 mongod`mongo::createProfileCollection(txn=0x0000000117bac8f8, db=0x00007f964060a550) + 1335 at introspect.cpp:185
    frame #10: 0x000000010e860b22 mongod`mongo::profile(txn=0x0000000117bac8f8, op=dbQuery) + 1570 at introspect.cpp:116
    frame #11: 0x000000010e8523b8 mongod`mongo::assembleResponse(txn=0x0000000117bac8f8, m=0x0000000117baac30, dbresponse=0x0000000117baaa00, remote=0x0000000110e10840) + 8376 at instance.cpp:623
    frame #12: 0x000000010e5cbd07 mongod`mongo::DBDirectClient::call(this=0x0000000117bab560, toSend=0x0000000117baac30, response=0x00007f964061dce8, assertOk=false, actualServer="localhost") + 263 at dbdirectclient.cpp:130
    frame #13: 0x000000010e2114de mongod`mongo::DBClientCursor::init(this=0x00007f964061dce0) + 302 at dbclientcursor.cpp:139
    frame #14: 0x000000010e1ded1a mongod`mongo::DBClientBase::query(this=0x0000000117bab560, ns="admin.system.roles", query=Query @ 0x0000000117bab0c0, nToReturn=0, nToSkip=0, fieldsToReturn=0x0000000117bab758, queryOptions=0, batchSize=0) + 474 at dbclient.cpp:1085
    frame #15: 0x000000010e5cc119 mongod`mongo::DBDirectClient::query(this=0x0000000117bab560, ns="admin.system.roles", query=<unavailable>, nToReturn=0, nToSkip=0, fieldsToReturn=0x0000000117bab758, queryOptions=0, batchSize=0) + 153 at dbdirectclient.cpp:156
    frame #16: 0x000000010e1dfae5 mongod`mongo::DBClientBase::query(this=0x0000000117bab560, f=function<void (mongo::DBClientCursorBatchIterator &)> @ 0x0000000117bab3b0, ns="admin.system.roles", query=<unavailable>, fieldsToReturn=0x0000000117bab758, queryOptions=0)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, mongo::Query, mongo::BSONObj const*, int) + 197 at dbclient.cpp:1128
    frame #17: 0x000000010e1df913 mongod`mongo::DBClientBase::query(this=0x0000000117bab560, f=function<void (const mongo::BSONObj &)> @ 0x0000000117bab530, ns="admin.system.roles", query=Query @ 0x0000000117bab4e0, fieldsToReturn=0x0000000117bab758, queryOptions=0)>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, mongo::Query, mongo::BSONObj const*, int) + 323 at dbclient.cpp:1117
    frame #18: 0x000000010e3049e0 mongod`mongo::AuthzManagerExternalStateMongod::query(this=0x00007f964044edc0, txn=0x0000000117bac8f8, collectionName=0x0000000110e0c398, query=0x0000000117bab768, projection=0x0000000117bab758, resultProcessor=0x0000000117babb50)> const&) + 272 at authz_manager_external_state_d.cpp:70
    frame #19: 0x000000010e305865 mongod`mongo::AuthzManagerExternalStateLocal::_initializeRoleGraph(this=0x00007f964044edc0, txn=0x0000000117bac8f8) + 933 at authz_manager_external_state_local.cpp:336
    frame #20: 0x000000010e3052eb mongod`mongo::AuthzManagerExternalStateLocal::initialize(this=0x00007f964044edc0, txn=0x0000000117bac8f8) + 59 at authz_manager_external_state_local.cpp:48
    frame #21: 0x000000010e2f1d06 mongod`mongo::AuthorizationManager::initialize(this=0x00007f964044eef0, txn=0x0000000117bac8f8) + 118 at authorization_manager.cpp:608
...



 Comments   
Comment by Githook User [ 20/Apr/16 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-23394: Prevent AuthorizationManager deadlock with profiling
Branch: v3.2
https://github.com/mongodb/mongo/commit/71cda84613a751b6bc60f4b8b502bc367c6efe07

Comment by Githook User [ 20/Apr/16 ]

Author:

{u'username': u'spencerjackson', u'name': u'Spencer Jackson', u'email': u'spencer.jackson@mongodb.com'}

Message: SERVER-23394: Prevent AuthorizationManager deadlock with profiling
Branch: master
https://github.com/mongodb/mongo/commit/c3eabfb346726648de7b00ce428b531993822803

Generated at Thu Feb 08 04:03:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.