[SERVER-72713] SessionWorkflow: taskRunner lease is destroyed too early Created: 11/Jan/23  Updated: 29/Oct/23  Resolved: 12/Jan/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.3.0-rc0

Type: Bug Priority: Major - P3
Reporter: Billy Donahue Assignee: Billy Donahue
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Service Arch 2023-01-23
Participants:
Linked BF Score: 118

 Description   

(Fix for BF-27075)

The _taskRunner data member of SessionWorkflow holds a claim on the worker thread so it can't be reused by the ServiceExecutor. This is released when a session is closing.

It is released too soon. Here's a patch:

@ src/mongo/transport/session_workflow.cpp:799 @ void SessionWorkflow::Impl::_cleanupExhaustResources() {
 
 void SessionWorkflow::Impl::_cleanupSession(const Status& status) {
     LOGV2_DEBUG(5127900, 2, "Ending session", "error"_attr = status);
-    _taskRunner = {};
     _cleanupExhaustResources();
+    _taskRunner = {};
     _sep->onClientDisconnect(client());
 }

The _cleanupExhaustResources function sends a killCursors command to the database. In mongod this is usually a very quick operation so the vulnerability is small. In a mongobridge configuration, this killCursors command, which is generated spontaneously inside the SessionWorkflow, has to be forwarded to the associated mongod, and this time window is much larger, causing a deadlock in the jstest ShardingTest constructor about 30% of the time!

The mechanics of the deadlock are complex. When setting up a new session, the ServiceExecutorSynchronous will lease a worker thread, creating a new one if none are idle. In this jstest, a worker happens to encounter a session disconnect during a getMore (exhaust) sequence. Due to this ordering bug, that worker appears to be idle and is immediately leased to an incoming connection, even though it is still performing the _cleanupExhuastResources command as part of cleanup (costly on mongobridge). The work of serving the new connection is placed on the task queue of this worker, but the worker never performs that task.

The worker should have perhaps still performed the queued task, which I could consider to be an underlying bug here! This SessionWorkflow fix is really perhaps just reducing a window of vulnerability to the underlying ServiceExecutor worker state transition bug. This investigation is pending in BF-27075.



 Comments   
Comment by Billy Donahue [ 12/Jan/23 ]


This isn't a full fix for the root cause of BF-27075 but should reduce its frequency of occurrence to effectively zero.

Comment by Githook User [ 12/Jan/23 ]

Author:

{'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}

Message: SERVER-72713 SessionWorkflow: delete _taskRunner only after cleanupExhaustResources
Branch: master
https://github.com/mongodb/mongo/commit/f4b9f38dc2a3451c9cac3529df86ad0f39946f6e

Generated at Thu Feb 08 06:22:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.