[SERVER-27831] Deadlock when listing collections on "local" database with replication enabled for KVCatalog-based storage engines without document locking Created: 27/Jan/17  Updated: 06/Dec/17  Resolved: 29/Mar/17

Status: Closed
Project: Core Server
Component/s: Storage
Affects Version/s: None
Fix Version/s: 3.4.7, 3.5.6

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Daniel Gottlieb (Inactive)
Resolution: Done Votes: 0
Labels: bkp, disabled-test, todo_in_code
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
is related to SERVER-21965 Deadlock between drop database and li... Closed
is related to SERVER-23994 Re-enable the concurrency and fuzzer ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v3.4
Steps To Reproduce:

Apply the following patch and run the create_database.js FSM workload against the ephemeralForTest storage engine.

python buildscripts/resmoke.py --executor=concurrency_replication jstests/concurrency/fsm_all_replication.js --storageEngine=ephemeralForTest

diff --git a/jstests/concurrency/fsm_libs/runner.js b/jstests/concurrency/fsm_libs/runner.js
index ec62924..7e33d1b 100644
--- a/jstests/concurrency/fsm_libs/runner.js
+++ b/jstests/concurrency/fsm_libs/runner.js
@@ -673,6 +673,7 @@ var runner = (function() {
                 bgThreadMgr.checkFailed(0);
 
                 var schedule = scheduleWorkloads(workloads, executionMode, executionOptions);
+                schedule = [ [ "jstests/concurrency/fsm_workloads/create_database.js" ] ];
                 printWorkloadSchedule(schedule, bgWorkloads);
 
                 schedule.forEach(function(workloads) {

Sprint: Storage 2017-03-27, Storage 2017-04-17
Participants:
Linked BF Score: 0

 Description   

Storage engines which use the KVCatalog, but do not support document-level concurrency (such as the ephemeralForTest storage engine) may experience deadlock because of the incompatible acquisition order of the resourceIdCatalogMetadata and the "local" database locks. This bug does not apply to the MMAPv1 or WiredTiger storage engines.

db.getSiblingDB("local").runCommand({listCollections: 1}):
  HOLD: Global, 1, MODE_IS
  HOLD: Global, 2, MODE_IS
  HOLD: Database, local, MODE_S
  WAIT ON: Metadata, resourceIdCatalogMetadata, MODE_S
 
db.getSiblingDB("CreateDatabase0").runCommand({create: "coll0"}):
  HOLD: Global, 1, MODE_IS
  HOLD: Global, 2, MODE_IX
  HOLD: Database, CreateDatabase0, MODE_X
  HOLD: Metadata, resourceIdCatalogMetadata, MODE_X
  WAIT ON: Database, local, MODE_IX

The resourceIdCatalogMetadata lock is acquired when calling KVCatalog::newCollection() as part of Database::createCollection(); however, the lock doesn't get released immediately after returning from there since it was acquired in MODE_X inside a WriteUnitOfWork causing shouldDelayUnlock() to return true. This leads to the client creating a collection to be holding the resourceIdCatalogMetadata lock while attempting to acquire a lock on the "local" database in order to write the corresponding oplog entry at the same time as the client listing collections on the "local" database to be holding a lock on the "local" database while attempting to acquire the resourceIdCatalogMetadata lock to get the individual collection options.


git version: ae04822985f2478c7da1e6821f5fc91b484b9555



 Comments   
Comment by Githook User [ 05/Jul/17 ]

Author:

{u'username': u'dgottlieb', u'name': u'Daniel Gottlieb', u'email': u'daniel.gottlieb@mongodb.com'}

Message: SERVER-27831 SERVER-28737: Push threadsafety responsibility inside of KVCatalog's RecordStore

RecordStores that don't implement document level locking are typically
protected from concurrent reads and writes. However one exception is the
RecordStore passed into KVCatalog. Previously, if
StorageEngine::supportsDocLocking was false, the KVCatalog would use
an additional lock that participated in two phase locking to ensure
reader-writer protection to the underlying record store (and more
specifically, delay releasing until any potential rollbacks were
processed). However, access to the catalog can happen anywhere and this
lock did not have a formally assigned acquisition time relative to other
locks resulting in potential deadlocks.

This patch forces the thread-safety requirement into the RecordStore.
Specifically, EphemeralForTest was changed to acquire a finer grained
mutex (i.e: does not participate in two-phase locking) to protect its
internal state. Now that the lock exists inside EphemeralForTest, it is
also able to grab the mutex when needed for the onCommit/onRollback
callbacks.

(cherry picked from commit 71a149b45c8bb019cbc8179f4a411be66bda2062)
(cherry picked from commit 99d1ad9e6ef9bd270f9f668966a71596a71f7f72)
Branch: v3.4
https://github.com/mongodb/mongo/commit/b188121303e27761b34bfb67be0732c346999168

Comment by Githook User [ 29/Mar/17 ]

Author:

{u'username': u'dgottlieb', u'name': u'Daniel Gottlieb', u'email': u'daniel.gottlieb@10gen.com'}

Message: SERVER-27831: Push threadsafety responsibility inside of KVCatalog's RecordStore

RecordStores that don't implement document level locking are typically
protected from concurrent reads and writes. However one exception is the
RecordStore passed into KVCatalog. Previously, if
StorageEngine::supportsDocLocking was false, the KVCatalog would use
an additional lock that participated in two phase locking to ensure
reader-writer protection to the underlying record store (and more
specifically, delay releasing until any potential rollbacks were
processed). However, access to the catalog can happen anywhere and this
lock did not have a formally assigned acquisition time relative to other
locks resulting in potential deadlocks.

This patch forces the thread-safety requirement into the RecordStore.
Specifically, EphemeralForTest was changed to acquire a finer grained
lock (i.e: does not participate in two-phase locking) to protect its
internal state. Now that the lock exists inside EphemeralForTest, it is
also able to grab the lock when needed for the onCommit/onRollback
callbacks.
Branch: master
https://github.com/mongodb/mongo/commit/71a149b45c8bb019cbc8179f4a411be66bda2062

Comment by Eric Milkie [ 28/Feb/17 ]

It's also possible to deadlock createCollection with itself, when creating collections in the local database, for the same reason that listing collections in the local database can trigger a deadlock: both operations hold the local database lock while trying to lock the resourceIdCatalogMetadata lock.

Generated at Thu Feb 08 04:16:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.