Details
-
Bug
-
Resolution: Fixed
-
Major - P3
-
None
-
None
-
None
-
Fully Compatible
-
ALL
-
Service Arch 2023-01-23
-
118
Description
(Fix for BF-27075)
The _taskRunner data member of SessionWorkflow holds a claim on the worker thread so it can't be reused by the ServiceExecutor. This is released when a session is closing.
It is released too soon. Here's a patch:
@ src/mongo/transport/session_workflow.cpp:799 @ void SessionWorkflow::Impl::_cleanupExhaustResources() { |
|
|
void SessionWorkflow::Impl::_cleanupSession(const Status& status) { |
LOGV2_DEBUG(5127900, 2, "Ending session", "error"_attr = status); |
- _taskRunner = {};
|
_cleanupExhaustResources();
|
+ _taskRunner = {};
|
_sep->onClientDisconnect(client());
|
}
|
The _cleanupExhaustResources function sends a killCursors command to the database. In mongod this is usually a very quick operation so the vulnerability is small. In a mongobridge configuration, this killCursors command, which is generated spontaneously inside the SessionWorkflow, has to be forwarded to the associated mongod, and this time window is much larger, causing a deadlock in the jstest ShardingTest constructor about 30% of the time!
The mechanics of the deadlock are complex. When setting up a new session, the ServiceExecutorSynchronous will lease a worker thread, creating a new one if none are idle. In this jstest, a worker happens to encounter a session disconnect during a getMore (exhaust) sequence. Due to this ordering bug, that worker appears to be idle and is immediately leased to an incoming connection, even though it is still performing the _cleanupExhuastResources command as part of cleanup (costly on mongobridge). The work of serving the new connection is placed on the task queue of this worker, but the worker never performs that task.
The worker should have perhaps still performed the queued task, which I could consider to be an underlying bug here! This SessionWorkflow fix is really perhaps just reducing a window of vulnerability to the underlying ServiceExecutor worker state transition bug. This investigation is pending in BF-27075.