[SERVER-58370] StringMap element use-after-free in RollbackImpl Created: 08/Jul/21  Updated: 29/Oct/23  Resolved: 08/Jul/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.0.4, 5.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Billy Donahue Assignee: Billy Donahue
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
is depended on by SERVER-51476 Upgrade Abseil to 20210324.1 Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v5.0
Sprint: Service Arch 2021-07-12
Participants:

 Description   

Latent bug exposed by running a patch build with upgraded Abseil.
Blocks abseil upgrade SERVER-51476.

https://logkeeper.mongodb.org/lobster/build/41e3896b888cdd98e03a025a5f67c92b/test/60e66134be07c4418a8af617#bookmarks=0%2C12156%2C12172%2C12173%2C12208%2C12260&l=1

[cpp_unit_test:db_repl_test] ==22468==ERROR: AddressSanitizer: heap-use-after-free on address 0x60600123fb38 at pc 0x7fb9b2644280 bp 0x7ffe2ddc9240 sp 0x7ffe2ddc9238
[cpp_unit_test:db_repl_test] READ of size 8 at 0x60600123fb38 thread T0
[cpp_unit_test:db_repl_test]     #0 0x7fb9b264427f in mongo::logv2::detail::NamedAttribute::NamedAttribute<long long>(char const*, long long const&) /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/logv2/attribute_storage.h:568:75
[cpp_unit_test:db_repl_test]     #1 0x7fb9b264427f in mongo::logv2::detail::AttributeStorage<mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&> >::AttributeStorage(mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&) /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/logv2/attribute_storage.h:596
[cpp_unit_test:db_repl_test]     #2 0x7fb9b2643e76 in mongo::logv2::detail::AttributeStorage<mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&> > mongo::logv2::detail::makeAttributeStorage<mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&> >(mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&) /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/logv2/attribute_storage.h:611:12
[cpp_unit_test:db_repl_test]     #3 0x7fb9b26438dd in void mongo::logv2::detail::doLogUnpacked<char [33], long long&, long long&, long long&>(int, mongo::logv2::LogSeverity const&, mongo::logv2::LogOptions const&, char const (&) [33], mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&) /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/logv2/log_detail.h:64:23
[cpp_unit_test:db_repl_test]     #4 0x7fb9b25f680b in void mongo::logv2::detail::doLogUnpacked<mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING, 33ul, long long&, long long&, long long&>(int, mongo::logv2::LogSeverity const&, mongo::logv2::LogOptions const&, mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING const&, char const (&) [33ul], mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&) /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/logv2/log_detail.h:77:5
[cpp_unit_test:db_repl_test]     #5 0x7fb9b25f66c8 in auto void mongo::logv2::detail::doLog<mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING, 33ul, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&> >(int, mongo::logv2::LogSeverity const&, mongo::logv2::LogOptions const&, mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING const&, char const (&) [33ul], mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&)::'lambda'(auto&&...)::operator()<mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&>(auto&&...) const /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/logv2/log_detail.h:123:13
[cpp_unit_test:db_repl_test]     #6 0x7fb9b25f66c8 in mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING std::__invoke_impl<void, void mongo::logv2::detail::doLog<mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING, 33ul, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&> >(int, mongo::logv2::LogSeverity const&, mongo::logv2::LogOptions const&, mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING const&, char const (&) [33ul], mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&)::'lambda'(auto&&...), mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&>(std::__invoke_other, void mongo::logv2::detail::doLog<mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING, 33ul, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&> >(int, mongo::logv2::LogSeverity const&, mongo::logv2::LogOptions const&, mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING const&, char const (&) [33ul], mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&)::'lambda'(auto&&...)&&, mongo::logv2::detail::NamedArg<long long&>&&, mongo::logv2::detail::NamedArg<long long&>&&, mongo::logv2::detail::NamedArg<long long&>&&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.ulc/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:60
[cpp_unit_test:db_repl_test]     #7 0x7fb9b25dad38 in decltype(auto) std::apply<void mongo::logv2::detail::doLog<mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING, 33ul, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&> >(int, mongo::logv2::LogSeverity const&, mongo::logv2::LogOptions const&, mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING const&, char const (&) [33ul], mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&)::'lambda'(auto&&...), std::tuple<mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&> >(mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING&&, std::tuple<mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&>&&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.ulc/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1687:14
[cpp_unit_test:db_repl_test]     #8 0x7fb9b25dad38 in void mongo::logv2::detail::doLog<mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING, 33ul, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&>, mongo::logv2::detail::NamedArg<long long&> >(int, mongo::logv2::LogSeverity const&, mongo::logv2::LogOptions const&, mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint)::$_16::operator()() const::FMT_COMPILE_STRING const&, char const (&) [33ul], mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&, mongo::logv2::detail::NamedArg<long long&> const&) /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/logv2/log_detail.h:121
[cpp_unit_test:db_repl_test]     #9 0x7fb9b25cd757 in mongo::repl::RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns(mongo::OperationContext*, mongo::repl::RollBackLocalOperations::RollbackCommonPoint) /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/db/repl/rollback_impl.cpp:586:5
[cpp_unit_test:db_repl_test]     #10 0x7fb9b25c8085 in mongo::repl::RollbackImpl::runRollback(mongo::OperationContext*) /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/db/repl/rollback_impl.cpp:241:5
[cpp_unit_test:db_repl_test]     #11 0x55aab2d593c3 in mongo::repl::(anonymous namespace)::RollbackImplObserverInfoTest::rollbackOps(std::__cxx11::list<std::pair<mongo::BSONObj, mongo::RecordId>, std::allocator<std::pair<mongo::BSONObj, mongo::RecordId> > > const&) /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/db/repl/rollback_impl_test.cpp:1883:27
[cpp_unit_test:db_repl_test]     #12 0x55aab2d5c9e2 in mongo::repl::(anonymous namespace)::UnitTest_SuiteNameRollbackImplObserverInfoTestTestNameRollbackRecordsNamespaceOfSingleOplogEntry::_doTest() /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/db/repl/rollback_impl_test.cpp:2153:5
[cpp_unit_test:db_repl_test]     #13 0x7fb9ad38341f in mongo::unittest::Test::run() /data/mci/67e0615db0e9449586187ebfd0cbe0b3/src/src/mongo/unittest/unittest.cpp:242:9
[cpp_unit_test:db_repl_test]     #14 0x55aab2d5bd41 in mongo::unittest::Test::RegistrationAgent<mongo::repl::(anonymous namespace)::UnitTest_SuiteNameRollbackImplObserverInfoTestTestNameRollbackRecordsNamespaceOfSingleOplogEntry>::RegistrationAgent(mongo::StringData, mongo::StringData, mongo::StringData)::'lambda'()::operator()() const /data/mci/a6bc3302eff4013565ce414462ee05e4/src/src/mongo/unittest/unittest.h:531:21
[cpp_unit_test:db_repl_test]     #15 0x55aab2d5bd41 in std::_Function_handler<void (), mongo::unittest::Test::RegistrationAgent<mongo::repl::(anonymous namespace)::UnitTest_SuiteNameRollbackImplObserverInfoTestTestNameRollbackRecordsNamespaceOfSingleOplogEntry>::RegistrationAgent(mongo::StringData, mongo::StringData, mongo::StringData)::'lambda'()>::_M_invoke(std::_Any_data const&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.ulc/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/std_function.h:297
[cpp_unit_test:db_repl_test]     #16 0x7fb9ad3865bf in mongo::unittest::Suite::run(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) /data/mci/67e0615db0e9449586187ebfd0cbe0b3/src/src/mongo/unittest/unittest.cpp:424:21
[cpp_unit_test:db_repl_test]     #17 0x7fb9ad38a3de in mongo::unittest::Suite::run(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) /data/mci/67e0615db0e9449586187ebfd0cbe0b3/src/src/mongo/unittest/unittest.cpp:493:26
[cpp_unit_test:db_repl_test]     #18 0x7fb9b9424b18 in main /data/mci/67e0615db0e9449586187ebfd0cbe0b3/src/src/mongo/unittest/unittest_main.cpp:122:19
[cpp_unit_test:db_repl_test]     #19 0x7fb998366bf6 in __libc_start_main /build/glibc-S9d2JN/glibc-2.27/csu/../csu/libc-start.c:310
[cpp_unit_test:db_repl_test]     #20 0x55aab21aea69 in _start (/data/mci/aae644862e10288648bcd0c4e63f08c0/src/build/install/bin/db_repl_test+0xc92a69)
[cpp_unit_test:db_repl_test] 



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 29/Sep/21 ]

Author:

{'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}

Message: SERVER-58370 dangling refs in RollbackImpl log statement

(cherry picked from commit 2fc4192641b6d90bcf25aea3a90ad868918eb306)
Branch: v5.0
https://github.com/mongodb/mongo/commit/7edcaa23a0b552f77cf6941de6501a0975584d53

Comment by Githook User [ 08/Jul/21 ]

Author:

{'name': 'Billy Donahue', 'email': 'billy.donahue@mongodb.com', 'username': 'BillyDonahue'}

Message: SERVER-58370 dangling refs in RollbackImpl log statement
Branch: master
https://github.com/mongodb/mongo/commit/2fc4192641b6d90bcf25aea3a90ad868918eb306

Comment by Billy Donahue [ 08/Jul/21 ]

Code Review: https://mongodbcr.appspot.com/812280001/

Comment by Billy Donahue [ 08/Jul/21 ]

The LOGV2 statement attaches 3 elements from a StringMap, obtained via operator[] , so they're references bound to _attr fields.
But because there are 3 of them, they can each cause an insert which can cause a rehash which can invalidate the previously obtained and bound element references!
What a mess!

https://github.com/mongodb/mongo/blob/master/src/mongo/db/repl/rollback_impl.cpp#L586-L592

    LOGV2(21599,
          "Rollback reverted {insert} insert operations, {update} update operations and {delete} "
          "delete operations.",
          "Rollback reverted command counts",
          "insert"_attr = _observerInfo.rollbackCommandCounts[kInsertCmdName],
          "update"_attr = _observerInfo.rollbackCommandCounts[kUpdateCmdName],
          "delete"_attr = _observerInfo.rollbackCommandCounts[kDeleteCmdName]);

Generated at Thu Feb 08 05:44:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.