-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Networking
-
Fully Compatible
-
ALL
-
-
Service Arch 2019-02-11, Service Arch 2019-02-25, Service Arch 2019-03-11, Service Arch 2019-03-25, Service Arch 2019-04-08, Service Arch 2019-04-22
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Found by billy.donahue while working on getting our vendored gperftools 2.5 building in C++17 mode.
When executor/connection_pool_test is built with libc++ and tcmalloc, it reports memory corruption, which was confirmed with ASAN:
=================================================================
==14437==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070000031c0 at pc 0x5557cd67be9f bp 0x7ffd033ff750 sp 0x7ffd033ff748
WRITE of size 8 at 0x6070000031c0 thread T0
#0 0x5557cd67be9e in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&) /usr/include/c++/v1/functional:1825:10
#1 0x5557cd67be9e in mongo::executor::connection_pool_test_details::TimerImpl::cancelTimeout() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:58
#2 0x5557cd686b7b in mongo::executor::connection_pool_test_details::TimerImpl::clear() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:64:16
#3 0x5557cd686b7b in mongo::executor::connection_pool_test_details::PoolImpl::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.h:166
#4 0x5557cd6c17d2 in mongo::executor::ConnectionPool::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:303:15
#5 0x5557cd6c1543 in mongo::executor::ConnectionPool::~ConnectionPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:299:5
#6 0x5557cd5f91c1 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:969:1
#7 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9
#8 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21
#9 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482
#10 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349
#11 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562
#12 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12
#13 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285
#14 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313
#15 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33
#16 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12
#17 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
#18 0x5557cd4c06d9 in _start (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x2b26d9)
0x6070000031c0 is located 48 bytes inside of 80-byte region [0x607000003190,0x6070000031e0)
freed by thread T0 here:
#0 0x5557cd5b9908 in operator delete(void*) (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x3ab908)
#1 0x5557cd6cb09d in std::__1::__shared_count::__release_shared() /usr/include/c++/v1/memory:3490:9
#2 0x5557cd6cb09d in std::__1::__shared_weak_count::__release_shared() /usr/include/c++/v1/memory:3532
#3 0x5557cd6cb09d in std::__1::shared_ptr<mongo::executor::ConnectionPool::TimerInterface>::~shared_ptr() /usr/include/c++/v1/memory:4468
#4 0x5557cd6cb09d in mongo::executor::ConnectionPool::SpecificPool::~SpecificPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:469
#5 0x5557cd6e0739 in std::__1::default_delete<mongo::executor::ConnectionPool::SpecificPool>::operator()(mongo::executor::ConnectionPool::SpecificPool*) const /usr/include/c++/v1/memory:2285:5
#6 0x5557cd6e0739 in std::__1::__shared_ptr_pointer<mongo::executor::ConnectionPool::SpecificPool*, std::__1::default_delete<mongo::executor::ConnectionPool::SpecificPool>, std::__1::allocator<mongo::executor::ConnectionPool::SpecificPool> >::__on_zero_shared() /usr/include/c++/v1/memory:3586
#7 0x5557cd6d83c9 in std::__1::__shared_count::__release_shared() /usr/include/c++/v1/memory:3490:9
#8 0x5557cd6d83c9 in std::__1::__shared_weak_count::__release_shared() /usr/include/c++/v1/memory:3532
#9 0x5557cd6d83c9 in std::__1::shared_ptr<mongo::executor::ConnectionPool::SpecificPool>::~shared_ptr() /usr/include/c++/v1/memory:4468
#10 0x5557cd6d83c9 in mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11::~$_11() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:946
#11 0x5557cd6d83c9 in std::__1::__compressed_pair_elem<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11, 0, false>::~__compressed_pair_elem() /usr/include/c++/v1/memory:2083
#12 0x5557cd6d83c9 in std::__1::__function::__func<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11, std::__1::allocator<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11>, void ()>::destroy() /usr/include/c++/v1/functional:1543
#13 0x5557cd67bc90 in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&) /usr/include/c++/v1/functional
#14 0x5557cd67bc90 in mongo::executor::connection_pool_test_details::TimerImpl::cancelTimeout() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:58
#15 0x5557cd686b7b in mongo::executor::connection_pool_test_details::TimerImpl::clear() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:64:16
#16 0x5557cd686b7b in mongo::executor::connection_pool_test_details::PoolImpl::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.h:166
#17 0x5557cd6c17d2 in mongo::executor::ConnectionPool::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:303:15
#18 0x5557cd6c1543 in mongo::executor::ConnectionPool::~ConnectionPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:299:5
#19 0x5557cd5f91c1 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:969:1
#20 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9
#21 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21
#22 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482
#23 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349
#24 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562
#25 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12
#26 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285
#27 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313
#28 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33
#29 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12
#30 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
previously allocated by thread T0 here:
#0 0x5557cd5b8b90 in operator new(unsigned long) (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x3aab90)
#1 0x5557cd68473a in std::__1::__unique_if<mongo::executor::connection_pool_test_details::TimerImpl>::__unique_single std::__1::make_unique<mongo::executor::connection_pool_test_details::TimerImpl, mongo::executor::connection_pool_test_details::PoolImpl*>(mongo::executor::connection_pool_test_details::PoolImpl*&&) /usr/include/c++/v1/memory:3078:28
#2 0x5557cd68473a in mongo::executor::connection_pool_test_details::PoolImpl::makeTimer() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:246
#3 0x5557cd6ca6c7 in mongo::executor::ConnectionPool::SpecificPool::SpecificPool(mongo::executor::ConnectionPool*, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:456:39
#4 0x5557cd6c531b in std::__1::__unique_if<mongo::executor::ConnectionPool::SpecificPool>::__unique_single std::__1::make_unique<mongo::executor::ConnectionPool::SpecificPool, mongo::executor::ConnectionPool*, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode&>(mongo::executor::ConnectionPool*&&, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode&) /usr/include/c++/v1/memory:3078:32
#5 0x5557cd6c531b in mongo::executor::ConnectionPool::get(mongo::HostAndPort const&, mongo::transport::ConnectSSLMode, mongo::Duration<std::__1::ratio<1l, 1000l> >) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:399
#6 0x5557cd6c4d65 in mongo::executor::ConnectionPool::get_forTest(mongo::HostAndPort const&, mongo::Duration<std::__1::ratio<1l, 1000l> >, std::__1::function<void (mongo::StatusWith<std::__1::unique_ptr<mongo::executor::ConnectionPool::ConnectionInterface, std::__1::function<void (mongo::executor::ConnectionPool::ConnectionInterface*)> > >)>) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:369:12
#7 0x5557cd5f8745 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:943:10
#8 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9
#9 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21
#10 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482
#11 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349
#12 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562
#13 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12
#14 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285
#15 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313
#16 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33
#17 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12
#18 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/v1/functional:1825:10 in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&)
Shadow bytes around the buggy address:
0x0c0e7fff85e0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
0x0c0e7fff85f0: fd fd fd fd fd fd fa fa fa fa fd fd fd fd fd fd
0x0c0e7fff8600: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
0x0c0e7fff8610: fd fd fa fa fa fa fd fd fd fd fd fd fd fd fd fd
0x0c0e7fff8620: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fa fa
=>0x0c0e7fff8630: fa fa fd fd fd fd fd fd[fd]fd fd fd fa fa fa fa
0x0c0e7fff8640: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa
0x0c0e7fff8650: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff8660: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff8670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c0e7fff8680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==14437==ABORTING