[SERVER-59888] Deadlock between BucketCatalog::_statesMutex and BucketCatalog::_idleMutex when expiring idle buckets Created: 10/Sep/21  Updated: 29/Oct/23  Resolved: 13/Sep/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.3, 5.1.0-rc0

Type: Bug Priority: Critical - P2
Reporter: Gregory Noma Assignee: Gregory Noma
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File 0001-SERVER-59888.patch    
Issue Links:
Backports
Related
is related to SERVER-59925 Stress test and fix deadlock in expir... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: Execution Team 2021-09-20
Participants:
Case:

 Description   

Consider two concurrent time-series inserts: one which inserts into an existing bucket, and one which expires an idle bucket in order to allocate a new one.

The former acquires BucketCatalog::_statesMutex inside of BucketCatalog::BucketAccess::_confirmStateForAcquiredBucket and then later acquires BucketCatalog::_idleMutex inside of BucketCatalog::_markBucketNotIdle.

The latter acquires BucketCatalog::_idleMutex inside of BucketCatalog::_expireIdleBuckets and then later acquires BucketCatalog::_statesMutex inside of BucketCatalog::_removeBucket.

Deadlock.

The full BucketCatalog stack traces to reach this deadlock are:

mongo::BucketCatalog::insert
mongo::BucketCatalog::BucketAccess::BucketAccess
mongo::BucketCatalog::BucketAccess::_findOpenBucketThenLock
mongo::BucketCatalog::BucketAccess::_confirmStateForAcquiredBucket
mongo::BucketCatalog::_markBucketNotIdle

and

mongo::BucketCatalog::insert
mongo::BucketCatalog::BucketAccess::BucketAccess
mongo::BucketCatalog::BucketAccess::_findOrCreateOpenBucketThenLock
mongo::BucketCatalog::BucketAccess::_create
mongo::BucketCatalog::_allocateBucket
mongo::BucketCatalog::_expireIdleBuckets
mongo::BucketCatalog::_removeBucket

Attached is a patch which reproduces the deadlock.



 Comments   
Comment by Githook User [ 14/Sep/21 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-59888 Ensure consistent acquisition order for `_statesMutex` and `_idleMutex` in the `BucketCatalog`

(cherry picked from commit 6bd9072c982bec88a61573ec723bce711a6c54a1)
Branch: v5.0
https://github.com/mongodb/mongo/commit/b20afc2d424ac2548d1a6c9eae979fba2cb42733

Comment by Githook User [ 13/Sep/21 ]

Author:

{'name': 'Gregory Noma', 'email': 'gregory.noma@gmail.com', 'username': 'gregorynoma'}

Message: SERVER-59888 Ensure consistent acquisition order for `_statesMutex` and `_idleMutex` in the `BucketCatalog`
Branch: master
https://github.com/mongodb/mongo/commit/6bd9072c982bec88a61573ec723bce711a6c54a1

Generated at Thu Feb 08 05:48:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.