[SERVER-37430] Destroy sharding task executors and AsyncRequestSenders after the PeriodicRunner is destroyed in mongod shutdown Created: 02/Oct/18  Updated: 08/Jan/24  Resolved: 03/Oct/18

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.6.8, 4.0.2, 4.1.3
Fix Version/s: 3.6.9, 4.0.4, 4.1.4

Type: Bug Priority: Major - P3
Reporter: Blake Oler Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-37335 Invariant failure _targeted.empty() i... Closed
is depended on by SERVER-37404 Terminate called on AsyncRequestsSend... Closed
is depended on by SERVER-37405 Exception in ProducerConsumerQueue us... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0, v3.6
Sprint: Sharding 2018-10-08
Participants:

 Description   

Problem Description

Some tasks created to use the PeriodicRunner make use of sharding task executors and AsyncRequestSenders, such as the LogicalSessionCache refresh or reap threads.

In shutdown on mongod, we destroy the Grid's task executors before we destroy the PeriodicRunner. This can lead to both attempted use of destroyed data, and shutdown exceptions when trying to call member functions on the task executors and AsyncRequestSenders.

Proposed Fix

The simple fix is to shut down the PeriodicRunner before shutting down Sharding-related machinery. We will place the call to shut down the PeriodicRunner here.

This fix has been verified to fix errors in an Evergreen patch.



 Comments   
Comment by Githook User [ 05/Oct/18 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-37430 Shut down PeriodicRunner before Sharding and Replication shutdown

(cherry picked from commit 860b392d9d3c006090a4c7fc3c6f3fa5460e5c5c)
Branch: v3.6
https://github.com/mongodb/mongo/commit/947477673298740bafae48cd865f06fd5b794eeb

Comment by Githook User [ 05/Oct/18 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-37430 Shut down PeriodicRunner before Sharding and Replication shutdown

(cherry picked from commit 860b392d9d3c006090a4c7fc3c6f3fa5460e5c5c)
Branch: v4.0
https://github.com/mongodb/mongo/commit/875a5fb980415a8b89e5c95d70fec4f448366652

Comment by Githook User [ 03/Oct/18 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-37430 Shut down PeriodicRunner before Sharding and Replication shutdown
Branch: master
https://github.com/mongodb/mongo/commit/860b392d9d3c006090a4c7fc3c6f3fa5460e5c5c

Comment by Andy Schwerin [ 02/Oct/18 ]

blake.oler, I don't know. sorry

Comment by Blake Oler [ 02/Oct/18 ]

schwerin what would be the quickest way to verify if it produces a cycle in the graph? I can say at the very least that I haven't seen any cycles created (proxy verification through no new failures), and I ran 25 patches with this new code today.

Comment by Andy Schwerin [ 02/Oct/18 ]

Does this produce a cycle in the shutdown graph? Is there anything that must be shutdown before destroying the periodic runner? I can't think of anything.

Comment by Misha Tyulenev [ 02/Oct/18 ]

lgtm blake.oler

Generated at Thu Feb 08 04:45:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.