[SERVER-38456] killSessionsLocalKillTransactions must filter for sessions with running transactions Created: 07/Dec/18  Updated: 29/Oct/23  Resolved: 07/Mar/19

Status: Closed
Project: Core Server
Component/s: Replication, Sharding
Affects Version/s: None
Fix Version/s: 4.1.9

Type: Bug Priority: Major - P3
Reporter: Tess Avitabile (Inactive) Assignee: Kaloian Manassiev
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Problem/Incident
Related
related to SERVER-57167 Prevent throwing on session creation ... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Sharding 2019-01-14, Sharding 2019-01-28, Sharding 2019-02-11, Sharding 2019-02-25, Sharding 2019-03-11
Participants:
Linked BF Score: 62

 Description   

In stepdown, we enqueue an RSTL X lock and then kill all running transactions by calling killSessionsLocalKillTransactions(). This visits every session in the session catalog, kills the operation that has the session checked out, waits for the session to be checked in, checks out the session, then aborts any running transaction. However, if the operation is a retryable write, then it can perform query yielding, which reacquires locks in an UninterruptibleLockGuard. An operation cannot be killed while it is in an UninterruptibleLockGuard, and it will not check the session back in until it has reacquired the lock (which it cannot, since there is an RSTL X lock enqueued). This problem also occurred when the Global X lock was used to guard state transitions.

I recommend solving this by changing the filter that killSessionsLocalKillTransactions() supplies to killSessionsAction() so that it only attempts to visit operations with running multi-document transactions. Multi-document transactions never yield locks, so they will not attempt to reacquire locks in an UninterruptibleLockGuard.



 Comments   
Comment by Githook User [ 07/Mar/19 ]

Author:

{'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}

Message: SERVER-38456 On shutdown, stepdown and FCV downgrade only kill sessions with multi-statement transactions
Branch: master
https://github.com/mongodb/mongo/commit/f7492fe762273c1fa41d68275407ca08d1d6a970

Comment by Tess Avitabile (Inactive) [ 11/Dec/18 ]

A retryable write can also scan many documents, looking for the first matching document to update. They would yield in that scan.

Comment by Siyuan Zhou [ 11/Dec/18 ]

The reads in retryable writes are reading transaction table and the oplog, right? They should be fairly cheap. Is disabling yielding an option for that?

Comment by Kaloian Manassiev [ 07/Dec/18 ]

Spoke with Tess in person and found out that only retryable writes check-out the session, regular operations outside of transaction don't.

Because of this, making all yield code paths check-in the session is a bit of an overkill and also goes against future plans for replication to not kill read operations.

The simplest solution would be for the killSessionsLocalKillTransactions() call to only consider TransactionParticipants which are running a session.

Comment by Kaloian Manassiev [ 07/Dec/18 ]

tess.avitabile, a retryable write operation is no different from a find and they both can yield and lead to the same uninterruptability, can't they? A solution where we inspect the type of TransactionParticipant on the session is also racy, because there is no guarantee whether we will catch the retryable write before or after it has yielded.

We had discussed previously about query yielding to check-in the session - would this work?

Another alternative that just occurred to me while talking to Siyuan was to have a different kind of uninterruptability, which is effectively "I am ok with yielding my place in the queue and letting other operations jump in-front of me, but I can't stop waiting". Effectively it will be a loop where we wait for lock reacquisition and if we get "Interrupted" we yield the locks and continue waiting. I would like to think about this a bit more, but what do you think about just checking in the session at yield time?

Generated at Thu Feb 08 04:49:00 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.