Loading...

Type: Bug
Resolution: Fixed
Priority: Major - P3
Fix Version/s: 4.1.11
Affects Version/s: None
Component/s: Networking
Labels:
- address-sanitizer

Backwards Compatibility:
Fully Compatible
Operating System:
ALL
Steps To Reproduce:
Hide

On commit 524cc45b29e023c9004f51f8a90c5bb7b0d7c169, build on Ubuntu 18.04 as follows:

python buildscripts/scons.py --dbg=on --opt=on --variables-files= CC=clang-6.0 CXX=clang++-6.0 --libc++ --disable-warnings-as-errors --sanitize=address --allocator=system -j24 build/optdebug/mongo/executor/connection_pool_test LIBS=ssl

And then run

export ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-6.0/bin/llvm-symbolizer export ASAN_OPTIONS=symbolize=1 ./build/optdebug/mongo/executor/connection_pool_test
Show
On commit 524cc45b29e023c9004f51f8a90c5bb7b0d7c169, build on Ubuntu 18.04 as follows: python buildscripts/scons.py --dbg=on --opt=on --variables-files= CC=clang-6.0 CXX=clang++-6.0 --libc++ --disable-warnings-as-errors --sanitize=address --allocator=system -j24 build/optdebug/mongo/executor/connection_pool_test LIBS=ssl And then run export ASAN_SYMBOLIZER_PATH=/usr/lib/llvm-6.0/bin/llvm-symbolizer export ASAN_OPTIONS=symbolize=1 ./build/optdebug/mongo/executor/connection_pool_test
Sprint:
Service Arch 2019-02-11, Service Arch 2019-02-25, Service Arch 2019-03-11, Service Arch 2019-03-25, Service Arch 2019-04-08, Service Arch 2019-04-22

Found by billy.donahue while working on getting our vendored gperftools 2.5 building in C++17 mode.

When executor/connection_pool_test is built with libc++ and tcmalloc, it reports memory corruption, which was confirmed with ASAN:

=================================================================
==14437==ERROR: AddressSanitizer: heap-use-after-free on address 0x6070000031c0 at pc 0x5557cd67be9f bp 0x7ffd033ff750 sp 0x7ffd033ff748
WRITE of size 8 at 0x6070000031c0 thread T0
    #0 0x5557cd67be9e in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&) /usr/include/c++/v1/functional:1825:10
    #1 0x5557cd67be9e in mongo::executor::connection_pool_test_details::TimerImpl::cancelTimeout() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:58
    #2 0x5557cd686b7b in mongo::executor::connection_pool_test_details::TimerImpl::clear() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:64:16
    #3 0x5557cd686b7b in mongo::executor::connection_pool_test_details::PoolImpl::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.h:166
    #4 0x5557cd6c17d2 in mongo::executor::ConnectionPool::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:303:15
    #5 0x5557cd6c1543 in mongo::executor::ConnectionPool::~ConnectionPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:299:5
    #6 0x5557cd5f91c1 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:969:1
    #7 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9
    #8 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21
    #9 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482
    #10 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349
    #11 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562
    #12 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12
    #13 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285
    #14 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313
    #15 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33
    #16 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12
    #17 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
    #18 0x5557cd4c06d9 in _start (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x2b26d9)

0x6070000031c0 is located 48 bytes inside of 80-byte region [0x607000003190,0x6070000031e0)
freed by thread T0 here:
    #0 0x5557cd5b9908 in operator delete(void*) (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x3ab908)
    #1 0x5557cd6cb09d in std::__1::__shared_count::__release_shared() /usr/include/c++/v1/memory:3490:9
    #2 0x5557cd6cb09d in std::__1::__shared_weak_count::__release_shared() /usr/include/c++/v1/memory:3532
    #3 0x5557cd6cb09d in std::__1::shared_ptr<mongo::executor::ConnectionPool::TimerInterface>::~shared_ptr() /usr/include/c++/v1/memory:4468
    #4 0x5557cd6cb09d in mongo::executor::ConnectionPool::SpecificPool::~SpecificPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:469
    #5 0x5557cd6e0739 in std::__1::default_delete<mongo::executor::ConnectionPool::SpecificPool>::operator()(mongo::executor::ConnectionPool::SpecificPool*) const /usr/include/c++/v1/memory:2285:5
    #6 0x5557cd6e0739 in std::__1::__shared_ptr_pointer<mongo::executor::ConnectionPool::SpecificPool*, std::__1::default_delete<mongo::executor::ConnectionPool::SpecificPool>, std::__1::allocator<mongo::executor::ConnectionPool::SpecificPool> >::__on_zero_shared() /usr/include/c++/v1/memory:3586
    #7 0x5557cd6d83c9 in std::__1::__shared_count::__release_shared() /usr/include/c++/v1/memory:3490:9
    #8 0x5557cd6d83c9 in std::__1::__shared_weak_count::__release_shared() /usr/include/c++/v1/memory:3532
    #9 0x5557cd6d83c9 in std::__1::shared_ptr<mongo::executor::ConnectionPool::SpecificPool>::~shared_ptr() /usr/include/c++/v1/memory:4468
    #10 0x5557cd6d83c9 in mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11::~$_11() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:946
    #11 0x5557cd6d83c9 in std::__1::__compressed_pair_elem<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11, 0, false>::~__compressed_pair_elem() /usr/include/c++/v1/memory:2083
    #12 0x5557cd6d83c9 in std::__1::__function::__func<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11, std::__1::allocator<mongo::executor::ConnectionPool::SpecificPool::updateStateInLock()::$_11>, void ()>::destroy() /usr/include/c++/v1/functional:1543
    #13 0x5557cd67bc90 in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&) /usr/include/c++/v1/functional
    #14 0x5557cd67bc90 in mongo::executor::connection_pool_test_details::TimerImpl::cancelTimeout() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:58
    #15 0x5557cd686b7b in mongo::executor::connection_pool_test_details::TimerImpl::clear() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:64:16
    #16 0x5557cd686b7b in mongo::executor::connection_pool_test_details::PoolImpl::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.h:166
    #17 0x5557cd6c17d2 in mongo::executor::ConnectionPool::shutdown() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:303:15
    #18 0x5557cd6c1543 in mongo::executor::ConnectionPool::~ConnectionPool() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:299:5
    #19 0x5557cd5f91c1 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:969:1
    #20 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9
    #21 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21
    #22 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482
    #23 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349
    #24 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562
    #25 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12
    #26 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285
    #27 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313
    #28 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33
    #29 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12
    #30 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

previously allocated by thread T0 here:
    #0 0x5557cd5b8b90 in operator new(unsigned long) (/home/andrew/Documents/10gen/dev/src/mongodb/build/cached/mongo/executor/connection_pool_test+0x3aab90)
    #1 0x5557cd68473a in std::__1::__unique_if<mongo::executor::connection_pool_test_details::TimerImpl>::__unique_single std::__1::make_unique<mongo::executor::connection_pool_test_details::TimerImpl, mongo::executor::connection_pool_test_details::PoolImpl*>(mongo::executor::connection_pool_test_details::PoolImpl*&&) /usr/include/c++/v1/memory:3078:28
    #2 0x5557cd68473a in mongo::executor::connection_pool_test_details::PoolImpl::makeTimer() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test_fixture.cpp:246
    #3 0x5557cd6ca6c7 in mongo::executor::ConnectionPool::SpecificPool::SpecificPool(mongo::executor::ConnectionPool*, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:456:39
    #4 0x5557cd6c531b in std::__1::__unique_if<mongo::executor::ConnectionPool::SpecificPool>::__unique_single std::__1::make_unique<mongo::executor::ConnectionPool::SpecificPool, mongo::executor::ConnectionPool*, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode&>(mongo::executor::ConnectionPool*&&, mongo::HostAndPort const&, mongo::transport::ConnectSSLMode&) /usr/include/c++/v1/memory:3078:32
    #5 0x5557cd6c531b in mongo::executor::ConnectionPool::get(mongo::HostAndPort const&, mongo::transport::ConnectSSLMode, mongo::Duration<std::__1::ratio<1l, 1000l> >) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:399
    #6 0x5557cd6c4d65 in mongo::executor::ConnectionPool::get_forTest(mongo::HostAndPort const&, mongo::Duration<std::__1::ratio<1l, 1000l> >, std::__1::function<void (mongo::StatusWith<std::__1::unique_ptr<mongo::executor::ConnectionPool::ConnectionInterface, std::__1::function<void (mongo::executor::ConnectionPool::ConnectionInterface*)> > >)>) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool.cpp:369:12
    #7 0x5557cd5f8745 in mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens::_doTest() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/executor/connection_pool_test.cpp:943:10
    #8 0x5557cd6adf1d in mongo::unittest::Test::run() /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:188:9
    #9 0x5557cd674288 in mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}::operator()() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:420:21
    #10 0x5557cd674288 in _ZNSt3__18__invokeIRZN5mongo8unittest5Suite3addINS1_8executor28connection_pool_test_details48UnitTest__ConnectionPoolTest__hostTimeoutHappensEEEvRKNS_12basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEEEUlvE_JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOSI_DpOSJ_ /usr/include/c++/v1/type_traits:4482
    #11 0x5557cd674288 in void std::__1::__invoke_void_return_wrapper<void>::__call<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&>(mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}&) /usr/include/c++/v1/__functional_base:349
    #12 0x5557cd674288 in std::__1::__function::__func<mongo::unittest::Suite::add<mongo::executor::connection_pool_test_details::UnitTest__ConnectionPoolTest__hostTimeoutHappens>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)::{lambda()#1}, std::__1::allocator<{lambda()#1}>, void ()>::operator()() /usr/include/c++/v1/functional:1562
    #13 0x5557cd6b12e3 in std::__1::function<void ()>::operator()() const /usr/include/c++/v1/functional:1916:12
    #14 0x5557cd6b12e3 in mongo::unittest::TestHolder::run() const /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.h:285
    #15 0x5557cd6b12e3 in mongo::unittest::Suite::run(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:313
    #16 0x5557cd6b53c7 in mongo::unittest::Suite::run(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest.cpp:373:33
    #17 0x5557cd6925d1 in main /home/andrew/Documents/10gen/dev/src/mongodb/src/mongo/unittest/unittest_main.cpp:112:12
    #18 0x7fd0bb0e1b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310

SUMMARY: AddressSanitizer: heap-use-after-free /usr/include/c++/v1/functional:1825:10 in std::__1::function<void ()>::operator=(std::__1::function<void ()>&&)
Shadow bytes around the buggy address:
  0x0c0e7fff85e0: fd fd fd fd fd fd fd fd fa fa fa fa fd fd fd fd
  0x0c0e7fff85f0: fd fd fd fd fd fd fa fa fa fa fd fd fd fd fd fd
  0x0c0e7fff8600: fd fd fd fd fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c0e7fff8610: fd fd fa fa fa fa fd fd fd fd fd fd fd fd fd fd
  0x0c0e7fff8620: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fa fa
=>0x0c0e7fff8630: fa fa fd fd fd fd fd fd[fd]fd fd fd fa fa fa fa
  0x0c0e7fff8640: 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa
  0x0c0e7fff8650: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff8660: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff8670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0e7fff8680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==14437==ABORTING

Details

Description

Attachments

Activity

People

Dates