Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-23082

race in ThreadPoolTaskExecutor between scheduleWorkAt() and cancel()

XMLWordPrintableJSON

    • Fully Compatible
    • ALL
    • 0

      Updated description:

      There is a potential race between the scheduled task in scheduleWorkAt() and cancel(). I was running into the scenario where the schedule task runs and makes it past the first isCanceled.load() check but is blocked waiting on the mutex because canceled() has acquired the lock at that moment and is in the process of setting the atomic and following through with
      invoking the callback with a CallbackCanceled status. And as soon as cancel() releases the lock, the task in scheduleWorkAt() acquires it and runs the callback a second time, deleting the same iterator from the inProgress std::list twice.

      -----------

      ASAN reported error: possible use-after-free error in ThreadPoolTaskExecutor::runCallback

      I first encountered this in a patch build with the AddressSanitizer enabled working on some changes to SyncSourceFeedback. To reproduce, re-apply the (reverted) commit 052c463 in an ASAN patch build with the following tasks:

      • replicasets
      • replicasets_WT
      • sharding
      • sharding_WT
       =================================================================
 ==70815==ERROR: AddressSanitizer: heap-use-after-free on address 0x60300023d448 at pc 0x000002a0ae59 bp 0x7f992221ef70 sp 0x7f992221ef68
 READ of size 8 at 0x60300023d448 thread T35
     #0 0x2a0ae58 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/shared_ptr_base.h:665:6
     #1 0x2a0ae58 in std::__shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/shared_ptr_base.h:914
     #2 0x2a0ae58 in std::_List_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >::~_List_node() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_list.h:106
     #3 0x2a0ae58 in void __gnu_cxx::new_allocator<std::_List_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::destroy<std::_List_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >(std::_List_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >*) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/ext/new_allocator.h:124
     #4 0x2a0ae58 in std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::_M_erase(std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_list.h:1700
     #5 0x2a0ae58 in std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::erase(std::_List_const_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/list.tcc:154
     #6 0x2a0ae58 in mongo::executor::ThreadPoolTaskExecutor::runCallback(std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>) /data/mci/src/src/mongo/executor/thread_pool_task_executor.cpp:471
     #7 0x2a0bfe4 in mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::unique_lock<std::mutex>)::$_4::operator()() const /data/mci/src/src/mongo/executor/thread_pool_task_executor.cpp:454:58
     #8 0x2a0bfe4 in std::_Function_handler<void (), mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::unique_lock<std::mutex>)::$_4>::_M_invoke(std::_Any_data const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/functional:2039
     #9 0x29f0f3e in std::function<void ()>::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/functional:2439:14
...

 0x60300023d448 is located 24 bytes inside of 32-byte region [0x60300023d430,0x60300023d450)
 freed by thread T35 here:
     #0 0xb7358b in operator delete(void*) (/data/mci/src/mongod+0xb7358b)
     #1 0x2a0a51a in __gnu_cxx::new_allocator<std::_List_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::deallocate(std::_List_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >*, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/ext/new_allocator.h:110:9
     #2 0x2a0a51a in std::_List_base<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::_M_put_node(std::_List_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >*) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_list.h:347
     #3 0x2a0a51a in std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::_M_erase(std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_list.h:1704
     #4 0x2a0a51a in std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::erase(std::_List_const_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/list.tcc:154
     #5 0x2a0a51a in mongo::executor::ThreadPoolTaskExecutor::runCallback(std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>) /data/mci/src/src/mongo/executor/thread_pool_task_executor.cpp:471
     #6 0x2a0bfe4 in mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::unique_lock<std::mutex>)::$_4::operator()() const /data/mci/src/src/mongo/executor/thread_pool_task_executor.cpp:454:58
     #7 0x2a0bfe4 in std::_Function_handler<void (), mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > const&, std::unique_lock<std::mutex>)::$_4>::_M_invoke(std::_Any_data const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/functional:2039
     #8 0x29f0f3e in std::function<void ()>::operator()() const /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/functional:2439:14
     #9 0x29f0f3e in mongo::executor::NetworkInterfaceThreadPool::consumeTasks(std::unique_lock<std::mutex>) /data/mci/src/src/mongo/executor/network_interface_thread_pool.cpp:159
     #10 0x29f1e15 in mongo::executor::NetworkInterfaceThreadPool::schedule(std::function<void ()>) /data/mci/src/src/mongo/executor/network_interface_thread_pool.cpp:112:9
...

 previously allocated by thread T35 here:
     #0 0xb7304b in operator new(unsigned long) (/data/mci/src/mongod+0xb7304b)
     #1 0x29ff586 in __gnu_cxx::new_allocator<std::_List_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/ext/new_allocator.h:104:27
     #2 0x29ff586 in std::_List_base<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::_M_get_node() /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_list.h:343
     #3 0x29ff586 in std::_List_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >* std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::_M_create_node<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >(std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_list.h:511
     #4 0x29ff586 in void std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::_M_insert<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >(std::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >, std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_list.h:1688
     #5 0x29ff586 in void std::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >::emplace_front<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >(std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/stl_list.h:998
     #6 0x29ff586 in mongo::executor::ThreadPoolTaskExecutor::makeSingletonWorkQueue(std::function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>, mongo::Date_t) /data/mci/src/src/mongo/executor/thread_pool_task_executor.cpp:409
     #7 0x2a02692 in mongo::executor::ThreadPoolTaskExecutor::scheduleWorkAt(mongo::Date_t, std::function<void (mongo::executor::TaskExecutor::CallbackArgs const&)> const&) /data/mci/src/src/mongo/executor/thread_pool_task_executor.cpp:246:15
     #8 0x22de6ca in mongo::repl::Reporter::_processResponseCallback(mongo::executor::TaskExecutor::RemoteCommandCallbackArgs const&) /data/mci/src/src/mongo/db/repl/reporter.cpp:292:17
...

 Thread T35 created by T31 here:
     #0 0xbc253f in __interceptor_pthread_create (/data/mci/src/mongod+0xbc253f)
     #1 0x7f993dcc8f08 in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbaf08)

 Thread T31 created by T17 here:
     #0 0xbc253f in __interceptor_pthread_create (/data/mci/src/mongod+0xbc253f)
     #1 0x7f993dcc8f08 in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbaf08)

 Thread T17 created by T12 here:
     #0 0xbc253f in __interceptor_pthread_create (/data/mci/src/mongod+0xbc253f)
     #1 0x7f993dcc8f08 in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbaf08)

 Thread T12 created by T0 here:
     #0 0xbc253f in __interceptor_pthread_create (/data/mci/src/mongod+0xbc253f)
     #1 0x7f993dcc8f08 in std::thread::_M_start_thread(std::shared_ptr<std::thread::_Impl_base>) (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0xbaf08)

 SUMMARY: AddressSanitizer: heap-use-after-free /usr/bin/../lib/gcc/x86_64-linux-gnu/4.9/../../../../include/c++/4.9/bits/shared_ptr_base.h:665 std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()
 Shadow bytes around the buggy address:
   0x0c068003fa30: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
   0x0c068003fa40: fd fd fa fa fd fd fd fa fa fa fd fd fd fa fa fa
   0x0c068003fa50: 00 00 00 00 fa fa 00 00 00 fa fa fa fd fd fd fa
   0x0c068003fa60: fa fa fd fd fd fa fa fa 00 00 00 06 fa fa fd fd
   0x0c068003fa70: fd fa fa fa 00 00 00 fa fa fa 00 00 00 fa fa fa
 =>0x0c068003fa80: fd fd fd fa fa fa fd fd fd[fd]fa fa 00 00 00 00
   0x0c068003fa90: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd
   0x0c068003faa0: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
   0x0c068003fab0: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa
   0x0c068003fac0: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
   0x0c068003fad0: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa
 Shadow byte legend (one shadow byte represents 8 application bytes):
   Addressable:           00
   Partially addressable: 01 02 03 04 05 06 07
   Heap left redzone:       fa
   Heap right redzone:      fb
   Freed heap region:       fd
   Stack left redzone:      f1
   Stack mid redzone:       f2
   Stack right redzone:     f3
   Stack partial redzone:   f4
   Stack after return:      f5
   Stack use after scope:   f8
   Global redzone:          f9
   Global init order:       f6
   Poisoned by user:        f7
   Container overflow:      fc
   ASan internal:           fe
 ==70815==ABORTING

        1. asan.txt
          56 kB
          Benety Goh

            Assignee:
            benety.goh@mongodb.com Benety Goh
            Reporter:
            benety.goh@mongodb.com Benety Goh
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: