[SERVER-58310] ThreadPoolTaskExecutor is memory unsafe when task cancellation occurs around the same time an exhaust network response is received Created: 06/Jul/21  Updated: 29/Oct/23  Resolved: 06/Jan/22

Status: Closed
Project: Core Server
Component/s: Networking
Affects Version/s: 4.4.0, 5.0.0-rc7
Fix Version/s: 4.4.14, 5.3.0, 5.0.7

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Amirsaman Memaripour
Resolution: Fixed Votes: 0
Labels: servicearch-wfbf-day
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
is related to SERVER-58221 Stress test for ThreadPoolTaskExecuto... Closed
is related to SERVER-62396 Add a non-deterministic variant for `... Backlog
is related to SERVER-45114 Connect AsyncDBClient exhaust functio... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.3, v5.0, v4.4
Sprint: Service Arch 2021-10-04, Service Arch 2021-12-13, Service Arch 2022-1-10, Service Arch 2022-1-24
Participants:
Linked BF Score: 20
Story Points: 3

 Description   

cbState->callback is modified without holding ThreadPoolTaskExecutor::_mutex in ThreadPoolTaskExecutor::runCallbackExhaust(). This is memory unsafe because cbState->callback is also modified in the exhaust codepath's RemoteCommandOnReplyFn upon task cancellation. This issue has been observed to cause server crashes when shutting down a replica set monitor due to a partially initialized TaskExecutor::CallbackFn callback being invoked.



 Comments   
Comment by Githook User [ 11/Mar/22 ]

Author:

{'name': 'Amirsaman Memaripour', 'email': 'amirsaman.memaripour@mongodb.com', 'username': 'samanca'}

Message: SERVER-58310 Always hold `_mutex` before modifying `CallbackState::callback`

(cherry picked from commit a9bec8a996b1d8fd6d3e28b200d8483f9a944bcb)
Branch: v4.4
https://github.com/mongodb/mongo/commit/12ce53745e54de01629cfc7d366c00a9d1dea058

Comment by Githook User [ 11/Mar/22 ]

Author:

{'name': 'Amirsaman Memaripour', 'email': 'amirsaman.memaripour@mongodb.com', 'username': 'samanca'}

Message: SERVER-58310 Always hold `_mutex` before modifying `CallbackState::callback`

(cherry picked from commit a9bec8a996b1d8fd6d3e28b200d8483f9a944bcb)
Branch: v5.0
https://github.com/mongodb/mongo/commit/51e18f041ca20eaf4821241939ea5a0216be45bb

Comment by Githook User [ 06/Jan/22 ]

Author:

{'name': 'Amirsaman Memaripour', 'email': 'amirsaman.memaripour@mongodb.com', 'username': 'samanca'}

Message: SERVER-58310 Always hold `_mutex` before modifying `CallbackState::callback`
Branch: master
https://github.com/mongodb/mongo/commit/a9bec8a996b1d8fd6d3e28b200d8483f9a944bcb

Comment by Max Hirschhorn [ 06/Jul/21 ]

It seems possible that a similar issue affects the non-exhaust ThreadPoolTaskExecutor::runCallback() codepath too. However, the task cancellation behavior is different so perhaps not.

Generated at Thu Feb 08 05:44:10 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.