[SERVER-43242] Deadlock involving commands acquiring global X lock and prepared transactions waiting for write concern Created: 09/Sep/19  Updated: 27/Oct/23  Resolved: 16/Jan/20

Status: Closed
Project: Core Server
Component/s: Replication
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Siyuan Zhou Assignee: Evin Roesle
Resolution: Gone away Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-43927 Make jstestfuzz_sharded and jstestfuz... Closed
Issue split
split to SERVER-44026 Remove global X lock for reIndex Closed
split to SERVER-44027 Remove global X lock for renameCollec... Closed
split to SERVER-44028 Remove global X lock for Cloner Closed
split to SERVER-44029 Remove global X lock for replSetResiz... Closed
split to SERVER-44030 Remove global X lock for mapReduce Closed
split to SERVER-44033 Only take the global write lock for a... Closed
split to SERVER-44037 Remove restartCatalog Closed
split to SERVER-33272 The DatabaseHolder::close() function ... Closed
Related
related to SERVER-45612 Remove the mapReduce + prepare testin... Closed
Operating System: ALL
Backport Requested:
v4.2
Sprint: Repl 2019-10-21, Repl 2019-11-04
Participants:
Linked BF Score: 21

 Description   

DropDatabase or any commands that acquire the global lock in X mode can be blocked by prepared transactions. The enqueued global X lock can block oplog queries which need the global IS lock. If these oplog queries and the data replication are needed by the prepared transaction's write concern, then the prepare transaction cannot make progress. Thus a deadlock occurs.

In reality, the coordinator may time out and decide to abort the transaction, but this could be an issue for testing.



 Comments   
Comment by Evin Roesle [ 16/Jan/20 ]

Gone Away in 4.4 featureCompatibilityVersion

Comment by Siyuan Zhou [ 20/Dec/19 ]

Reopening the ticket since mapreduce can still cause the deaklock.

Comment by James Wahlin [ 15/Oct/19 ]

We have a project in 4.4 to deprecate mapReduce

To be clear on the mapReduce time frame, we are currently working on replacing the command internals with an implementation that uses the aggregation framework. Once we are done with our project (during the 4.3 dev cycle) we will delete the legacy mongos implementation. The legacy mongod implementation, which is the part that can take a global X locks, will remain in place until we branch 4.4 and will be used by mixed-version clusters during 4.2->4.4 upgrade. It will be deleted early in the 4.5 dev cycle.

Comment by Lingzhi Deng [ 04/Oct/19 ]

Is the work of this ticket just to work around the issue in tests? Making oplog special is probably less error-prone in the long run?

Comment by Esha Maharishi (Inactive) [ 16/Sep/19 ]

judah.schvimer, hm, we could have it use a fixed short timeout that matches what is used elsewhere in sharding, like 30 or 60 seconds. I don't have all the context about this deadlock, so it was just a suggestion that would prevent tests from hanging forever.

Comment by Judah Schvimer [ 10/Sep/19 ]

esha.maharishi, what maxTimeMS would you include? If it's the transaction timeout, then that's set to (essentially) infinite in our tests.

Comment by Esha Maharishi (Inactive) [ 10/Sep/19 ]

judah.schvimer, siyuan.zhou, how about we make the coordinator send maxTimeMS in all prepareTransaction requests?

Comment by Judah Schvimer [ 10/Sep/19 ]

I'm not sure how to work around this in our tests besides blacklisting important test coverage, relaxing the expectations that transactions commit (and reducing the timeout), or implicitly adding a maxTimeMS to all commands that acquire a Global X Lock.

Comment by Siyuan Zhou [ 10/Sep/19 ]

Confirmed with esha.maharishi that the coordinator will time out on the prepareTransaction command after 1 minute by default and write an abort decision. The coordinator will then send the abortTransaction command to all nodes. Since waiting for write concern doesn't hold the session checked out and RSTL lock isn't involved. The abortTransaction should go through successfully and resolve the issue. If we don't fix this deadlock, I'd hope we can have tests for this automatic resolution.

Comment by Eric Milkie [ 10/Sep/19 ]

I actually think it would be fine to work around these in our tests. We have a project in 4.4 to deprecate mapReduce, usage of the cloner is almost gone, applyOps is internal, and reIndex can be fixed to not take a global lock.

Comment by Judah Schvimer [ 10/Sep/19 ]

reIndex, resizeOplog, mapReduce, the "Cloner" (so anything that is used for), and applyOps all take a Global X Lock. I don't think we can fix all of these (especially not mapReduce).

The urgency is definitely lower since it won't be a permanent deadlock and none of these commands are very common. That said, I don't think it's something we want to just work around in our tests. I would be pro oplog queries being more special. milkie, what do you think of that approach?

Comment by Eric Milkie [ 10/Sep/19 ]

SERVER-33272 will change dropDatabase to only acquire a DB lock. We can backport that to 4.2. There are still other commands that take a global lock, I believe.

Comment by Siyuan Zhou [ 09/Sep/19 ]

milkie and geert.bosch, did we remove the requirement of global X lock for dropDatabases? Do we still have other cases that can take a global lock in master and 4.2? We could remove all global X lock by external commands or make oplog queries even more special by requiring no lock or its own lock.

Generated at Thu Feb 08 05:02:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.