[SERVER-62951] Deadlock on restarting the node with a prepared transaction acquiring the global lock Created: 25/Jan/22 Updated: 06/Dec/22 |
|
| Status: | Open |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Moustafa Maher | Assignee: | Backlog - Replication Team |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||
| Assigned Teams: |
Replication
|
||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Linked BF Score: | 12 | ||||||||||||||||||||
| Description |
|
Acquiring the global lock in the initAndListen function while initialization mongod after starting up replication is dangerous as we might reconstruct a prepared transaction which acquire the global lock and keep it, which will cause a deadlock.
You can repro it by adding a sleep before this line and run jstests/replsets/ddl_op_behind_transaction_fails_in_shutdown.js |
| Comments |
| Comment by Max Hirschhorn [ 03/Feb/22 ] | ||||||||||||||||||||||||||||||||||||
|
I confirmed the hang only happens with {nodes: 1}. I attempted both with a single-voting node replica set and with a 3-voting node replica set. Sending this over to the Replication team because it doesn't appear this hang can happen in a production setting. I feel like it is surprising a mongod can transition to member state PRIMARY while _initAndListen() hasn't even started accepting incoming connections.
| ||||||||||||||||||||||||||||||||||||
| Comment by Moustafa Maher [ 02/Feb/22 ] | ||||||||||||||||||||||||||||||||||||
|
daniel.gottlieb The analysis of max.hirschhorn is correct! | ||||||||||||||||||||||||||||||||||||
| Comment by Max Hirschhorn [ 31/Jan/22 ] | ||||||||||||||||||||||||||||||||||||
daniel.gottlieb, it looks like the transactions are put into the prepared state synchronously at startup by ReplicationCoordinatorImpl::_startLoadLocalConfig(). However, locks for transactions are released rather than stashed when applying a transaction as a secondary ( What adding the sleep enables to happen is the node transitions to member state PRIMARY and reacquires locks for the prepared transaction. I'm not really sure where the best place to move creating the oplog view for tenant migrations is. Would we want to do it before starting up the ReplicationCoordinatorImpl? | ||||||||||||||||||||||||||||||||||||
| Comment by Daniel Gottlieb (Inactive) [ 25/Jan/22 ] | ||||||||||||||||||||||||||||||||||||
|
m.maher, can you comment why the sleep matters? Are we preparing transactions in a background thread at startup? I feel deadlocks at startup should be deterministic. |