[SERVER-51134] Restore invariant in ReplicaSetAwareServiceRegistry destructor that _services is empty Created: 24/Sep/20  Updated: 29/Oct/23  Resolved: 28/Sep/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.8.0

Type: Task Priority: Major - P3
Reporter: Spencer Brody (Inactive) Assignee: Kevin Pulo
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-49243 Add optional currentOp output for Pri... Closed
Backwards Compatibility: Fully Compatible
Sprint: Sharding 2020-10-05
Participants:

 Description   

SERVER-49243 removed an invariant in RSASR that all previously registered services have been unregistered by the time the registry is destroyed. This was removed because the changes from SERVER-49243 caused this invariant to start firing in mongo_embedded_test. We should try to understand what actually caused the invariant to start firing, fix the issue, and restore the invariant.



 Comments   
Comment by Githook User [ 28/Sep/20 ]

Author:

{'name': 'Kevin Pulo', 'email': 'kevin.pulo@mongodb.com', 'username': 'devkev'}

Message: SERVER-51134 ensure that RSASR has no services when destroyed, and embedded must runGlobalDeinitializers() later
Branch: master
https://github.com/mongodb/mongo/commit/7cac84e82eec3e9292cae02f5d85c2cacc30c4a4

Comment by Kevin Pulo [ 25/Sep/20 ]

Actually I think you've hit the nail on the head - the deinitializers are supposed to run very late, basically just before the process exits. Every other context (eg. unittest's main()) does this. But destroying the ServiceContext is Real Work, so it's improper for embedded to run the deinitializers before doing that. Moving the call to be after the ServiceContext is destroyed avoids triggering the RSASR invariant.

Comment by Spencer Brody (Inactive) [ 24/Sep/20 ]

So this code is getting way out of my area of expertise, but as best as I can tell from stepping through GDB, the problem is that embedded test calls runGlobalDeinitializers(), which then results in ConstructorDestructorActions for ReplicaSetAwareService getting erased from the static registeredConstructorActions list. That means that when it destroys the global service context, the destructor actions for that service context no longer exist, so the work to deregister registered ReplicaSetAwareServices never happens.

I don't really know why this is happening or what the right fix is. Below is a stack trace from GDB showing how the call to runGlobalDeinitializers() results in the registeredContructorAction being erased:

#0  mongo::ServiceContext::ConstructorActionRegisterer::<lambda(mongo::DeinitializerContext*)>::operator()(mongo::DeinitializerContext *) const (__closure=0x7fffdb6ad020, context=0x7fffffffccc3) at src/mongo/db/service_context.cpp:470
#1  0x00007ffff3be97b0 in std::_Function_handler<mongo::Status(mongo::DeinitializerContext*), mongo::ServiceContext::ConstructorActionRegisterer::ConstructorActionRegisterer(std::__cxx11::string, std::vector<std::__cxx11::basic_string<char> >, std::vector<std::__cxx11::basic_string<char> >, mongo::ServiceContext::ConstructorAction, mongo::ServiceContext::DestructorAction)::<lambda(mongo::DeinitializerContext*)> >::_M_invoke(const std::_Any_data &, mongo::DeinitializerContext *&&) (__functor=...,
    __args#0=@0x7fffffffcc68: 0x7fffffffccc3) at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/include/c++/8.2.0/bits/std_function.h:283
#2  0x00007ffff75219e0 in std::function<mongo::Status (mongo::DeinitializerContext*)>::operator()(mongo::DeinitializerContext*) const (this=0x7fffdb6ad020, __args#0=0x7fffffffccc3)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/include/c++/8.2.0/bits/std_function.h:687
#3  0x00007ffff7520c3a in mongo::Initializer::executeDeinitializers (this=0x7ffff7798360 <mongo::getGlobalInitializer()::theGlobalInitializer>) at src/mongo/base/initializer.cpp:101
#4  0x00007ffff7520df8 in mongo::runGlobalDeinitializers () at src/mongo/base/initializer.cpp:123
#5  0x00007ffff3d7cff8 in mongo::embedded::shutdown (srvContext=0x7fffdb295520) at src/mongo/embedded/embedded.cpp:177
#6  0x00007ffff780a898 in mongo::(anonymous namespace)::ServiceContextDestructor::operator() (this=0x7fffdb2ab570, serviceContext=0x7fffdb295520) at src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:142
#7  0x00007ffff780c5b3 in std::unique_ptr<mongo::ServiceContext, mongo::(anonymous namespace)::ServiceContextDestructor>::~unique_ptr (this=0x7fffdb2ab570, __in_chrg=<optimized out>)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/include/c++/8.2.0/bits/unique_ptr.h:274
#8  0x00007ffff7812dd6 in mongo_embedded_v1_instance::~mongo_embedded_v1_instance (this=0x7fffdb2ab560, __in_chrg=<optimized out>) at src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:151
#9  0x00007ffff7819438 in std::default_delete<mongo_embedded_v1_instance>::operator() (this=0x7fffdb4158f8, __ptr=0x7fffdb2ab560) at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/include/c++/8.2.0/bits/unique_ptr.h:81
#10 0x00007ffff7819d17 in std::unique_ptr<mongo_embedded_v1_instance, std::default_delete<mongo_embedded_v1_instance> >::reset (this=0x7fffdb4158f8, __p=0x7fffdb2ab560)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/include/c++/8.2.0/bits/unique_ptr.h:382
#11 0x00007ffff781695f in std::unique_ptr<mongo_embedded_v1_instance, std::default_delete<mongo_embedded_v1_instance> >::operator=(decltype(nullptr)) (this=0x7fffdb4158f8)
    at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/include/c++/8.2.0/bits/unique_ptr.h:318
#12 0x00007ffff780b5f6 in mongo::(anonymous namespace)::instance_destroy (db=0x7fffdb2ab560) at src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:352
#13 0x00007ffff780c252 in <lambda()>::operator()(void) const (__closure=0x7fffffffd050) at src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:489
#14 0x00007ffff780cc45 in mongo::enterCXXImpl<mongo::StatusForAPI<mongo_embedded_v1_error>, mongo_embedded_v1_instance_destroy(mongo_embedded_v1_instance*, mongo_embedded_v1_status*)::<lambda()>, void>::call<mongo_embedded_v1_instance_destroy(mongo_embedded_v1_instance*, mongo_embedded_v1_status*)::<lambda()> >(<lambda()> &&, mongo::StatusForAPI<mongo_embedded_v1_error> &, const mongo::ReentrancyGuard &&) (function=..., status=...) at src/mongo/embedded/api_common.h:156
#15 0x00007ffff780c844 in mongo::enterCXX<mongo::StatusForAPI<mongo_embedded_v1_error>, mongo_embedded_v1_instance_destroy(mongo_embedded_v1_instance*, mongo_embedded_v1_status*)::<lambda()> >(mongo::StatusForAPI<mongo_embedded_v1_error> *, <lambda()> &&) (
    statusPtr=0x7fffdb64fda0, c=...) at src/mongo/embedded/api_common.h:203
#16 0x00007ffff780c2a8 in mongo_embedded_v1_instance_destroy (db=0x7fffdb2ab560, statusPtr=0x7fffdb64fda0) at src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:489
#17 0x000055555559e7c4 in (anonymous namespace)::MongodbCAPITest::tearDown (this=0x7fffffffd390) at src/mongo/embedded/mongo_embedded/mongo_embedded_test.cpp:138
#18 0x00007ffff7a49901 in mongo::unittest::(anonymous namespace)::UnitTestEnvironment::~UnitTestEnvironment (this=0x7fffffffd348, __in_chrg=<optimized out>) at src/mongo/unittest/unittest.cpp:182
#19 0x00007ffff7a49ac7 in mongo::unittest::Test::run (this=0x7fffffffd390) at src/mongo/unittest/unittest.cpp:241
#20 0x00005555555ac348 in mongo::unittest::Test::RegistrationAgent<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameCreateAndDestroyDB>::<lambda()>::operator()(void) const (this=0x7fffdb6fa060) at src/mongo/unittest/unittest.h:530
#21 0x00005555555b0647 in std::_Function_handler<void(), mongo::unittest::Test::RegistrationAgent<T>::RegistrationAgent(mongo::StringData, mongo::StringData, mongo::StringData) [with T = (anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameCreateAndDestroyDB]::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/include/c++/8.2.0/bits/std_function.h:297
#22 0x00007ffff7a5cc30 in std::function<void ()>::operator()() const (this=0x7fffdb6fa060) at /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.U0D/include/c++/8.2.0/bits/std_function.h:687
#23 0x00007ffff7a4b591 in mongo::unittest::Suite::run (this=0x7fffdb6db570, filter=..., fileNameFilter=..., runsPerTest=1) at src/mongo/unittest/unittest.cpp:423
#24 0x00007ffff7a4c58b in mongo::unittest::Suite::run (suites=..., filter=..., fileNameFilter=..., runsPerTest=1) at src/mongo/unittest/unittest.cpp:492
#25 0x00005555555abe0d in main (argc=1, argv=0x7fffffffdd18) at src/mongo/embedded/mongo_embedded/mongo_embedded_test.cpp:771

Generated at Thu Feb 08 05:24:36 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.