[SERVER-43242] Deadlock involving commands acquiring global X lock and prepared transactions waiting for write concern Created: 09/Sep/19 Updated: 27/Oct/23 Resolved: 16/Jan/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Siyuan Zhou | Assignee: | Evin Roesle |
| Resolution: | Gone away | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v4.2
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | Repl 2019-10-21, Repl 2019-11-04 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 21 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
DropDatabase or any commands that acquire the global lock in X mode can be blocked by prepared transactions. The enqueued global X lock can block oplog queries which need the global IS lock. If these oplog queries and the data replication are needed by the prepared transaction's write concern, then the prepare transaction cannot make progress. Thus a deadlock occurs. In reality, the coordinator may time out and decide to abort the transaction, but this could be an issue for testing. |
| Comments |
| Comment by Evin Roesle [ 16/Jan/20 ] |
|
Gone Away in 4.4 featureCompatibilityVersion |
| Comment by Siyuan Zhou [ 20/Dec/19 ] |
|
Reopening the ticket since mapreduce can still cause the deaklock. |
| Comment by James Wahlin [ 15/Oct/19 ] |
To be clear on the mapReduce time frame, we are currently working on replacing the command internals with an implementation that uses the aggregation framework. Once we are done with our project (during the 4.3 dev cycle) we will delete the legacy mongos implementation. The legacy mongod implementation, which is the part that can take a global X locks, will remain in place until we branch 4.4 and will be used by mixed-version clusters during 4.2->4.4 upgrade. It will be deleted early in the 4.5 dev cycle. |
| Comment by Lingzhi Deng [ 04/Oct/19 ] |
|
Is the work of this ticket just to work around the issue in tests? Making oplog special is probably less error-prone in the long run? |
| Comment by Esha Maharishi (Inactive) [ 16/Sep/19 ] |
|
judah.schvimer, hm, we could have it use a fixed short timeout that matches what is used elsewhere in sharding, like 30 or 60 seconds. I don't have all the context about this deadlock, so it was just a suggestion that would prevent tests from hanging forever. |
| Comment by Judah Schvimer [ 10/Sep/19 ] |
|
esha.maharishi, what maxTimeMS would you include? If it's the transaction timeout, then that's set to (essentially) infinite in our tests. |
| Comment by Esha Maharishi (Inactive) [ 10/Sep/19 ] |
|
judah.schvimer, siyuan.zhou, how about we make the coordinator send maxTimeMS in all prepareTransaction requests? |
| Comment by Judah Schvimer [ 10/Sep/19 ] |
|
I'm not sure how to work around this in our tests besides blacklisting important test coverage, relaxing the expectations that transactions commit (and reducing the timeout), or implicitly adding a maxTimeMS to all commands that acquire a Global X Lock. |
| Comment by Siyuan Zhou [ 10/Sep/19 ] |
|
Confirmed with esha.maharishi that the coordinator will time out on the prepareTransaction command after 1 minute by default and write an abort decision. The coordinator will then send the abortTransaction command to all nodes. Since waiting for write concern doesn't hold the session checked out and RSTL lock isn't involved. The abortTransaction should go through successfully and resolve the issue. If we don't fix this deadlock, I'd hope we can have tests for this automatic resolution. |
| Comment by Eric Milkie [ 10/Sep/19 ] |
|
I actually think it would be fine to work around these in our tests. We have a project in 4.4 to deprecate mapReduce, usage of the cloner is almost gone, applyOps is internal, and reIndex can be fixed to not take a global lock. |
| Comment by Judah Schvimer [ 10/Sep/19 ] |
|
reIndex, resizeOplog, mapReduce, the "Cloner" (so anything that is used for), and applyOps all take a Global X Lock. I don't think we can fix all of these (especially not mapReduce). The urgency is definitely lower since it won't be a permanent deadlock and none of these commands are very common. That said, I don't think it's something we want to just work around in our tests. I would be pro oplog queries being more special. milkie, what do you think of that approach? |
| Comment by Eric Milkie [ 10/Sep/19 ] |
|
|
| Comment by Siyuan Zhou [ 09/Sep/19 ] |
|
milkie and geert.bosch, did we remove the requirement of global X lock for dropDatabases? Do we still have other cases that can take a global lock in master and 4.2? We could remove all global X lock by external commands or make oplog queries even more special by requiring no lock or its own lock. |