[SERVER-38456] killSessionsLocalKillTransactions must filter for sessions with running transactions Created: 07/Dec/18 Updated: 29/Oct/23 Resolved: 07/Mar/19 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Replication, Sharding |
| Affects Version/s: | None |
| Fix Version/s: | 4.1.9 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Tess Avitabile (Inactive) | Assignee: | Kaloian Manassiev |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Sprint: | Sharding 2019-01-14, Sharding 2019-01-28, Sharding 2019-02-11, Sharding 2019-02-25, Sharding 2019-03-11 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Linked BF Score: | 62 | ||||||||||||||||
| Description |
|
In stepdown, we enqueue an RSTL X lock and then kill all running transactions by calling killSessionsLocalKillTransactions(). This visits every session in the session catalog, kills the operation that has the session checked out, waits for the session to be checked in, checks out the session, then aborts any running transaction. However, if the operation is a retryable write, then it can perform query yielding, which reacquires locks in an UninterruptibleLockGuard. An operation cannot be killed while it is in an UninterruptibleLockGuard, and it will not check the session back in until it has reacquired the lock (which it cannot, since there is an RSTL X lock enqueued). This problem also occurred when the Global X lock was used to guard state transitions. I recommend solving this by changing the filter that killSessionsLocalKillTransactions() supplies to killSessionsAction() so that it only attempts to visit operations with running multi-document transactions. Multi-document transactions never yield locks, so they will not attempt to reacquire locks in an UninterruptibleLockGuard. |
| Comments |
| Comment by Githook User [ 07/Mar/19 ] |
|
Author: {'name': 'Kaloian Manassiev', 'username': 'kaloianm', 'email': 'kaloian.manassiev@mongodb.com'}Message: |
| Comment by Tess Avitabile (Inactive) [ 11/Dec/18 ] |
|
A retryable write can also scan many documents, looking for the first matching document to update. They would yield in that scan. |
| Comment by Siyuan Zhou [ 11/Dec/18 ] |
|
The reads in retryable writes are reading transaction table and the oplog, right? They should be fairly cheap. Is disabling yielding an option for that? |
| Comment by Kaloian Manassiev [ 07/Dec/18 ] |
|
Spoke with Tess in person and found out that only retryable writes check-out the session, regular operations outside of transaction don't. Because of this, making all yield code paths check-in the session is a bit of an overkill and also goes against future plans for replication to not kill read operations. The simplest solution would be for the killSessionsLocalKillTransactions() call to only consider TransactionParticipants which are running a session. |
| Comment by Kaloian Manassiev [ 07/Dec/18 ] |
|
tess.avitabile, a retryable write operation is no different from a find and they both can yield and lead to the same uninterruptability, can't they? A solution where we inspect the type of TransactionParticipant on the session is also racy, because there is no guarantee whether we will catch the retryable write before or after it has yielded. We had discussed previously about query yielding to check-in the session - would this work? Another alternative that just occurred to me while talking to Siyuan was to have a different kind of uninterruptability, which is effectively "I am ok with yielding my place in the queue and letting other operations jump in-front of me, but I can't stop waiting". Effectively it will be a loop where we wait for lock reacquisition and if we get "Interrupted" we yield the locks and continue waiting. I would like to think about this a bit more, but what do you think about just checking in the session at yield time? |