[SERVER-67732] mongod shutdown procedure doesn't shutdown the Grid::getExecutorPool() Created: 01/Jul/22  Updated: 29/Oct/23  Resolved: 06/Apr/23

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 7.0.0-rc0, 6.0.7, 5.0.19

Type: Bug Priority: Major - P3
Reporter: Sergi Mateo Bellido Assignee: Antonio Fuschetto
Resolution: Fixed Votes: 0
Labels: shardingemea-qw
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
related to SERVER-78971 Skip shutting down Grid::getExecutorP... Closed
is related to SERVER-67077 TransactionCoordinator uses both the ... Open
is related to SERVER-51316 Grid executor pool should be shut dow... Closed
is related to SERVER-73588 Suppress memory leak notification abo... Closed
Assigned Teams:
Sharding EMEA
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v6.3, v6.0, v5.0, v4.4
Sprint: Sharding EMEA 2022-07-25, Sharding EMEA 2022-12-26, Sharding EMEA 2023-01-09, Sharding EMEA 2023-01-23, Sharding EMEA 2023-02-06, Sharding EMEA 2023-04-17
Participants:
Linked BF Score: 35
Story Points: 1

 Description   

This could lead us to some memory leaks.

Note that the mongos shutdown procedure shutdowns the ExecutorPool.



 Comments   
Comment by Githook User [ 12/May/23 ]

Author:

{'name': 'Antonio Fuschetto', 'email': 'antonio.fuschetto@mongodb.com', 'username': 'afuschetto'}

Message: SERVER-67732 mongod shutdown procedure doesn't shutdown the Grid::getExecutorPool()
Branch: v5.0
https://github.com/mongodb/mongo/commit/35535a04d68d2d6fb9431f7861d60a1c5aa2ee60

Comment by Githook User [ 12/May/23 ]

Author:

{'name': 'Antonio Fuschetto', 'email': 'antonio.fuschetto@mongodb.com', 'username': 'afuschetto'}

Message: SERVER-67732 mongod shutdown procedure doesn't shutdown the Grid::getExecutorPool()
Branch: v6.0
https://github.com/mongodb/mongo/commit/5d90aeee4cf51dd4a7d6a42e965640848b86ba34

Comment by Liubov Molchanova [ 18/Apr/23 ]

Requesting a backport for v6.3 (reproduced in BFG-1897995 and some others)

Comment by Githook User [ 05/Apr/23 ]

Author:

{'name': 'Antonio Fuschetto', 'email': 'antonio.fuschetto@mongodb.com', 'username': 'afuschetto'}

Message: SERVER-67732 mongod shutdown procedure doesn't shutdown the Grid::getExecutorPool()
Branch: master
https://github.com/mongodb/mongo/commit/d1e8dd39698c5f1f7def7ac6c262aa588cc0ed6e

Comment by Antonio Fuschetto [ 30/Mar/23 ]

Since early January, LSAN has started highlighting memory leaks on many different threads, for example:

  • BFG-1739969: mongo::TransactionCoordinator::_done
  • BFG-1824813: mongo::TransactionRouter::Router::_resetRouterStateForStartTransaction
  • BFG-1825225: mongo::TransactionCoordinatorCatalog::_remove
  • BFG-1826533: mongo::CatalogCache::CollectionCache::_lookupCollection
  • BFG-1827567: mongo::Balancer::_mainThread

We should understand why we have so many warnings from January since these have always been there. Did we change the LSAN's configuration? In this scenario, we cannot apply the same strategy used for BF-25304, as in that case I changed the LSAN's configuration to suppress the memory leak warnings in one specific function.

I see no other options except 1) shutdown the Grid's fixed executor or 2) suppress these warnings at mongo at mongod shutdown. In the past, we evaluated the solution 1 and it turned out that the cost of analyzing and fixing all possible deadlocks would be too high to justify the effort.

On BF-27421, I asked the Service Architecture team to identify a workable solution to suppress these warning at level of LSAN's configuration.

Generated at Thu Feb 08 06:08:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.