[SERVER-16067] Contention on lockmgr buckets in MMAPv1 Created: 11/Nov/14  Updated: 04/Dec/14  Resolved: 03/Dec/14

Status: Closed
Project: Core Server
Component/s: Concurrency
Affects Version/s: 2.7.8
Fix Version/s: 2.8.0-rc2

Type: Bug Priority: Major - P3
Reporter: Davide Italiano Assignee: Geert Bosch
Resolution: Done Votes: 0
Labels: 28qa
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File lockmgrcontention.png    
Issue Links:
Related
related to SERVER-16065 Long flush pauses in MMAPv1 Closed
Tested
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

Reducing contention on GroupCommit mutex (SERVER-15950) exposed another bottleneck. The attached Vtune lock profiler analysis shows that the second most evident hot lock is the one that protect the lockmgr bucket.
Callstack:

Waiting Call Stack
1 of 10: 99.4% (549.400s of 552.974s)
 
mongod!lock+0xa - mutex.h:164
mongod!scoped_lock+0 - mutex.h:170
mongod!mongo::LockManager::lock+0x47 - lock_mgr_new.cpp:405
mongod!mongo::LockerImpl<(bool)1>::lockImpl+0x264 - lock_state.cpp:708
mongod!mongo::LockerImpl<(bool)1>::lock+0x27 - lock_state.cpp:422
mongod!mongo::LockerImpl<(bool)1>::endWriteUnitOfWork+0xfc - lock_state.cpp:407
mongod!mongo::WriteUnitOfWork::commit+0x3b - operation_context.h:176
mongod!singleInsert+0x1bf - batch_executor.cpp:1110
mongod!insertOne+0x1a0 - batch_executor.cpp:1038
mongod!mongo::WriteBatchExecutor::execOneInsert+0x5f - batch_executor.cpp:1063
mongod!mongo::WriteBatchExecutor::execInserts+0x22c - batch_executor.cpp:860
mongod!mongo::WriteBatchExecutor::bulkExecute+0x33 - batch_executor.cpp:754
mongod!mongo::WriteBatchExecutor::executeBatch+0x3a4 - batch_executor.cpp:263
mongod!mongo::WriteCmd::run+0x168 - write_commands.cpp:143
mongod!mongo::_execCommand+0x33 - dbcommands.cpp:1160
mongod!mongo::Command::execCommand+0xc50 - dbcommands.cpp:1374
mongod!mongo::_runCommands+0x1ef - dbcommands.cpp:1450
mongod!runCommands+0x23 - new_find.cpp:131
mongod!mongo::newRunQuery+0xff9 - new_find.cpp:551
mongod!receivedQuery+0x1f2 - instance.cpp:220
mongod!mongo::assembleResponse+0x9d0 - instance.cpp:393
mongod!mongo::MyMessageHandler::process+0xdf - db.cpp:185
mongod!mongo::PortMessageServer::handleIncomingMsg+0x420 -
message_server_port.cpp:234
libpthread-2.18.so!start_thread+0xc2 - [Unknown]:[Unknown]
libc-2.18.so!__clone+0x6c - [Unknown]:[Unknown]

and:

Waiting Call Stack
1 of 8: 99.9% (486.046s of 486.639s)
 
mongod!lock+0xa - mutex.h:164
mongod!scoped_lock+0 - mutex.h:170
mongod!mongo::LockManager::unlock+0x3b - lock_mgr_new.cpp:521
mongod!mongo::LockerImpl<(bool)1>::_unlockImpl+0xb0 - lock_state.cpp:718
mongod!mongo::LockerImpl<(bool)1>::unlock+0x1c3 - lock_state.cpp:516
mongod!mongo::LockerImpl<(bool)1>::endWriteUnitOfWork+0xcc - lock_state.cpp:403
mongod!mongo::WriteUnitOfWork::commit+0x3b - operation_context.h:176
mongod!singleInsert+0x1bf - batch_executor.cpp:1110
mongod!insertOne+0x1a0 - batch_executor.cpp:1038
mongod!mongo::WriteBatchExecutor::execOneInsert+0x5f - batch_executor.cpp:1063
mongod!mongo::WriteBatchExecutor::execInserts+0x22c - batch_executor.cpp:860
mongod!mongo::WriteBatchExecutor::bulkExecute+0x33 - batch_executor.cpp:754
mongod!mongo::WriteBatchExecutor::executeBatch+0x3a4 - batch_executor.cpp:263
mongod!mongo::WriteCmd::run+0x168 - write_commands.cpp:143
mongod!mongo::_execCommand+0x33 - dbcommands.cpp:1160
mongod!mongo::Command::execCommand+0xc50 - dbcommands.cpp:1374
mongod!mongo::_runCommands+0x1ef - dbcommands.cpp:1450
mongod!runCommands+0x23 - new_find.cpp:131
mongod!mongo::newRunQuery+0xff9 - new_find.cpp:551
mongod!receivedQuery+0x1f2 - instance.cpp:220
mongod!mongo::assembleResponse+0x9d0 - instance.cpp:393
mongod!mongo::MyMessageHandler::process+0xdf - db.cpp:185
mongod!mongo::PortMessageServer::handleIncomingMsg+0x420 -
message_server_port.cpp:234
libpthread-2.18.so!start_thread+0xc2 - [Unknown]:[Unknown]
libc-2.18.so!__clone+0x6c - [Unknown]:[Unknown]

kaloian.manassiev proposes to partition Global/Flush/DB locks to reduce this contention.



 Comments   
Comment by Githook User [ 02/Dec/14 ]

Author:

{u'username': u'GeertBosch', u'name': u'Geert Bosch', u'email': u'geert@mongodb.com'}

Message: SERVER-16067: Partition intent locks in the lock manager
Branch: master
https://github.com/mongodb/mongo/commit/6cab2a29c7dcb56ce132c200321bd1424f977f55

Comment by Githook User [ 24/Nov/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-16067 Simplify LockHead mode count tracking

Instead of using enumerations, use an increment/decrement calls in order
to avoid condition checking where it's not needed.

Inline all LockHead methods.
Branch: master
https://github.com/mongodb/mongo/commit/b452b55fd032a4e63d9ed25c60ecc5781e0a2b27

Comment by Githook User [ 24/Nov/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-16067 Move LockRequestList to a separate header file
Branch: master
https://github.com/mongodb/mongo/commit/5bcaa163b3fb6054cbedca1c20bd4ce98ddf1eed

Comment by Githook User [ 14/Nov/14 ]

Author:

{u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'}

Message: SERVER-16067 Split new lock acquisition and conversion into two methods

This makes it clear where we do acquisitions and conversions and we can do
less work and decision making while holding the Lock Manager's bucket
mutex.
Branch: master
https://github.com/mongodb/mongo/commit/0962abcb99b52b6a9d4d03e7980569a7933394b3

Generated at Thu Feb 08 03:39:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.