-
Type:
Improvement
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: None
-
Labels:None
-
Query Optimization
-
Fully Compatible
-
Service Arch 2023-12-11
To reproduce the crash, make a change to a golden file in ABTTranslationTest, such as src/mongo/db/test_output/pipeline/abt/a_b_t_translation_test/in_translation.txt. Then run ninja +abt_translation_test (or build/install/bin/db_pipeline_test --filter InTranslation in this example) and observe a crash.
This harms developer productivity as I am not able to generate all diffs for golden tests, since the test runner crashes on the first diff.
I did a preliminary investigation and found the following:
The GoldenTestContextBase class works by comparing the expected and observed log files in its destructor. If there is a diff, it will throw a TestAssertionFailureException. I believe the intent is for the unittest runner to catch this exception.
Here is stack trace that we observe in the repro described above:
#0 0x0000fffff7cc47cc in __cxa_throw () from /home/ubuntu/mongo/build/install/bin/../lib/libtcmalloc_minimal.so #1 0x0000ffffdfab6f58 in mongo::unittest::GoldenTestContext::onError (this=..., message=..., actualStr=..., expectedStr=...) at src/mongo/unittest/golden_test.cpp:83 #2 0x0000aaaaabf87d30 in _ZZN5mongo8unittest17GoldenTestContextC1EPKNS0_16GoldenTestConfigEPKNS0_8TestInfoEbENKUlDpRKT_E_clIJNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESJ_N5boost8optionalISJ_EEEEEDaSB_ (this=..., args=..., args=..., args=...) at src/mongo/unittest/golden_test.h:65 #3 0x0000aaaaabf87cdc in _ZSt13__invoke_implIvRZN5mongo8unittest17GoldenTestContextC1EPKNS1_16GoldenTestConfigEPKNS1_8TestInfoEbEUlDpRKT_E_JRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESM_RKN5boost8optionalISK_EEEET_St14__invoke_otherOT0_DpOT1_ (__f=..., __args=..., __args=..., __args=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:61 #4 0x0000aaaaabf87c2c in _ZSt10__invoke_rIvRZN5mongo8unittest17GoldenTestContextC1EPKNS1_16GoldenTestConfigEPKNS1_8TestInfoEbEUlDpRKT_E_JRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESM_RKN5boost8optionalISK_EEEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EEST_E4typeEOSU_DpOSV_ (__fn=..., __args=..., __args=..., __args=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:111 #5 0x0000aaaaabf87ae4 in _ZNSt17_Function_handlerIFvRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_RKN5boost8optionalIS5_EEEZN5mongo8unittest17GoldenTestContextC1EPKNSF_16GoldenTestConfigEPKNSF_8TestInfoEbEUlDpRKT_E_E9_M_invokeERKSt9_Any_dataS7_S7_SC_ (__functor=..., __args=..., __args=..., __args=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/std_function.h:290 #6 0x0000ffffdf8bf210 in std::function<void (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const&) const (this=..., __args=..., __args=..., __args=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/std_function.h:590 #7 0x0000ffffdf8bcaec in mongo::unittest::GoldenTestContextBase::failResultMismatch (this=..., actualStr=..., expectedStr=..., message=...) at src/mongo/unittest/golden_test_base.cpp:211 #8 0x0000ffffdf8bc838 in mongo::unittest::GoldenTestContextBase::verifyOutput (this=...) at src/mongo/unittest/golden_test_base.cpp:191 #9 0x0000aaaaabf825a4 in mongo::unittest::GoldenTestContextBase::~GoldenTestContextBase (this=...) at src/mongo/unittest/golden_test_base.h:162 #10 0x0000aaaaabf82568 in mongo::unittest::GoldenTestContext::~GoldenTestContext (this=...) at src/mongo/unittest/golden_test.h:48 #11 0x0000aaaaabf823dc in std::default_delete<mongo::unittest::GoldenTestContext>::operator() (this=..., __ptr=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/unique_ptr.h:85 #12 0x0000aaaaabf82288 in std::unique_ptr<mongo::unittest::GoldenTestContext, std::default_delete<mongo::unittest::GoldenTestContext> >::~unique_ptr (this=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/unique_ptr.h:361 #13 0x0000aaaaabf82300 in mongo::optimizer::ABTGoldenTestFixture::~ABTGoldenTestFixture (this=...) at src/mongo/db/query/optimizer/utils/unit_test_pipeline_utils.h:75 #14 0x0000aaaaabf92edc in mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation::~UnitTest_SuiteNameABTTranslationTestTestNameInTranslation (this=...) at src/mongo/db/pipeline/abt/abt_translation_test.cpp:86 #15 0x0000aaaaabf92e08 in mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}::operator()() const (this=...) at src/mongo/unittest/framework.h:346 #16 0x0000aaaaabf92d8c in std::__invoke_impl<void, mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}&>(std::__invoke_other, mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}&) (__f=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:61 #17 0x0000aaaaabf92d30 in std::__invoke_r<void, mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}&>(mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}&) (__fn=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:111 #18 0x0000aaaaabf92c24 in std::_Function_handler<void (), mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/std_function.h:290 #19 0x0000fffff276eb40 in std::function<void ()>::operator()() const (this=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/std_function.h:590 #20 0x0000ffffdfacae6c in mongo::unittest::Suite::run (this=..., filter=..., fileNameFilter=..., runsPerTest=...) at src/mongo/unittest/unittest.cpp:467 #21 0x0000ffffdfacbb70 in mongo::unittest::Suite::run (suites=..., filter=..., fileNameFilter=..., runsPerTest=...) at src/mongo/unittest/unittest.cpp:536 #22 0x0000fffff7bfed34 in main (argc=..., argv=...) at src/mongo/unittest/unittest_main.cpp:138
Immediately after the exception is thrown, some stack unwinding occurs and std::terminate is called. Here is the stack trace after that:
#0 __pthread_kill_implementation (threadid=..., signo=..., no_tid=...) at ./nptl/pthread_kill.c:44 #1 0x0000ffffcdb5f254 in __pthread_kill_internal (signo=..., threadid=...) at ./nptl/pthread_kill.c:78 #2 0x0000ffffcdb1a67c in __GI_raise (sig=...) at ../sysdeps/posix/raise.c:26 #3 0x0000ffffd015e268 in mongo::breakpoint () at src/mongo/util/debugger.cpp:70 #4 0x0000ffffd0106d08 in mongo::(anonymous namespace)::myTerminate () at src/mongo/util/signal_handlers_synchronous.cpp:276 #5 0x0000fffff7cc464c in __cxxabiv1::__terminate(void (*)()) () from /home/ubuntu/mongo/build/install/bin/../lib/libtcmalloc_minimal.so #6 0x0000fffff7cc46a0 in std::terminate() () from /home/ubuntu/mongo/build/install/bin/../lib/libtcmalloc_minimal.so #7 0x0000aaaaabf1c580 in __clang_call_terminate () #8 0x0000aaaaabf822a0 in std::unique_ptr<mongo::unittest::GoldenTestContext, std::default_delete<mongo::unittest::GoldenTestContext> >::~unique_ptr (this=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/unique_ptr.h:359 #9 0x0000aaaaabf82300 in mongo::optimizer::ABTGoldenTestFixture::~ABTGoldenTestFixture (this=...) at src/mongo/db/query/optimizer/utils/unit_test_pipeline_utils.h:75 #10 0x0000aaaaabf92edc in mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation::~UnitTest_SuiteNameABTTranslationTestTestNameInTranslation (this=...) at src/mongo/db/pipeline/abt/abt_translation_test.cpp:86 #11 0x0000aaaaabf92e08 in mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}::operator()() const (this=...) at src/mongo/unittest/framework.h:346 #12 0x0000aaaaabf92d8c in std::__invoke_impl<void, mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}&>(std::__invoke_other, mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}&) (__f=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:61 #13 0x0000aaaaabf92d30 in std::__invoke_r<void, mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}&>(mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}&) (__fn=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/invoke.h:111 #14 0x0000aaaaabf92c24 in std::_Function_handler<void (), mongo::unittest::Test::RegistrationAgent<mongo::optimizer::(anonymous namespace)::UnitTest_SuiteNameABTTranslationTestTestNameInTranslation>::RegistrationAgent(mongo::unittest::TestInfo const*)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/std_function.h:290 #15 0x0000fffff276eb40 in std::function<void ()>::operator()() const (this=...) at /opt/mongodbtoolchain/revisions/69f4f0673ffcb290ce2307560a4883ecf2ad138c/stow/gcc-v4.liy/lib/gcc/aarch64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/std_function.h:590 #16 0x0000ffffdfacae6c in mongo::unittest::Suite::run (this=..., filter=..., fileNameFilter=..., runsPerTest=...) at src/mongo/unittest/unittest.cpp:467 #17 0x0000ffffdfacbb70 in mongo::unittest::Suite::run (suites=..., filter=..., fileNameFilter=..., runsPerTest=...) at src/mongo/unittest/unittest.cpp:536 #18 0x0000fffff7bfed34 in main (argc=..., argv=...) at src/mongo/unittest/unittest_main.cpp:138
I believe what is occurring is that the unique_ptr that is being destructed here, has a destructor marked as noexcept, which invokes std::terminate when the exception is thrown.
All other uses of GoldenTestContext (outside of ABTGoldenTestFixture) do not put GoldenTestContext in a unique_ptr (example where GoldenTestContext is only put on the stack).
It seems like GoldenTestContext is doing a potentially dangerous thing by throwing an exception in its destructor, assuming that there are no other destructors in the call stack which are marked noexcept (the standard says all destructors are implicitly noexcept, that's why GoldenTestContextBase had to explicitly mark itself as noexcept(false)).
This ticket is to rewrite ABTGoldenTestFixture in such a way so that it doesn't maintain a unique_ptr to a GoldenTestContext which causes this destructor problem.