Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-69419

Race condition in setting/accessing LocalOplogInfo::_oplog CollectionPtr

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Replication
    • ALL
    • 24

      The race condition is around acquireOplogCollectionForLogging() attempting to set LocalOplogInfo::_oplog at the same time OplogCapMaintainerThread::_deleteExcessDocuments() is attempting to access it. Forwarding to repl.

      [js_test:audit_read_from_sharded_secondaries] d21290| ==================
      [js_test:audit_read_from_sharded_secondaries] d21290| WARNING: ThreadSanitizer: data race (pid=27563)
      [js_test:audit_read_from_sharded_secondaries] d21290|   Read of size 8 at 0x7ba80003b0e0 by thread T30:
      [js_test:audit_read_from_sharded_secondaries] d21290|     #0 mongo::CollectionPtr::operator bool() const /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/collection.h:788:34 (mongod+0x23c70c6)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #1 mongo::OplogCapMaintainerThread::_deleteExcessDocuments() /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/storage/oplog_cap_maintainer_thread.cpp:75 (mongod+0x23c70c6)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::OplogCapMaintainerThread::run() /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/storage/oplog_cap_maintainer_thread.cpp:114:14 (mongod+0x23c7e8d)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::BackgroundJob::jobBody() /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/util/background.cpp:162:5 (mongod+0x73fd08a)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #4 mongo::BackgroundJob::go()::$_0::operator()() const /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/util/background.cpp:188:31 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #5 void std::__invoke_impl<void, mongo::BackgroundJob::go()::$_0>(std::__invoke_other, mongo::BackgroundJob::go()::$_0&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:60 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #6 std::__invoke_result<mongo::BackgroundJob::go()::$_0>::type std::__invoke<mongo::BackgroundJob::go()::$_0>(mongo::BackgroundJob::go()::$_0&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:95 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #7 decltype(auto) std::__apply_impl<mongo::BackgroundJob::go()::$_0, std::tuple<> >(mongo::BackgroundJob::go()::$_0&&, std::tuple<>&&, std::integer_sequence<unsigned long>) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1678 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #8 decltype(auto) std::apply<mongo::BackgroundJob::go()::$_0, std::tuple<> >(mongo::BackgroundJob::go()::$_0&&, std::tuple<>&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1687 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #9 mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()::operator()() /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/stdx/thread.h:194 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #10 mongo::BackgroundJob::go()::$_0 std::__invoke_impl<void, mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()>(std::__invoke_other, mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:60 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #11 std::__invoke_result<mongo::BackgroundJob::go()::$_0>::type std::__invoke<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()>(mongo::BackgroundJob::go()::$_0&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:95 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #12 decltype(std::__invoke(_S_declval<0ul>())) std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/thread:244 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #13 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()> >::operator()() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/thread:253 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()> > >::_M_run() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/thread:196 (mongod+0x73ff8dc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #15 execute_native_thread_routine <null> (mongod+0x793948e)
      [js_test:audit_read_from_sharded_secondaries] d21290|
      [js_test:audit_read_from_sharded_secondaries] d21290|   Previous write of size 8 at 0x7ba80003b0e0 by main thread:
      [js_test:audit_read_from_sharded_secondaries] d21290|     #0 mongo::CollectionPtr::operator=(mongo::CollectionPtr&&) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/collection.cpp:52:56 (mongod+0x701cfb2)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #1 mongo::LocalOplogInfo::setCollection(mongo::CollectionPtr const&) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/local_oplog_info.cpp:71:12 (mongod+0x48c8dec)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::repl::acquireOplogCollectionForLogging(mongo::OperationContext*) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/oplog.cpp:2081:33 (mongod+0x3f3ae3a)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::repl::ReplicationCoordinatorImpl::startup(mongo::OperationContext*, mongo::StorageEngine::LastShutdownState) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/replication_coordinator_impl.cpp:947:9 (mongod+0x24c89c2)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #4 mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:720:20 (mongod+0x22ba1a1)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #5 mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:863:16 (mongod+0x22b0421)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #6 mongo::mongod_main(int, char**) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1575 (mongod+0x22b0421)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #7 main /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/db/mongod.cpp:47:22 (mongod+0x22acc1e)
      [js_test:audit_read_from_sharded_secondaries] d21290|
      [js_test:audit_read_from_sharded_secondaries] d21290|   Location is heap block of size 23921 at 0x7ba800036000 allocated by main thread:
      [js_test:audit_read_from_sharded_secondaries] d21290|     #0 operator new[](unsigned long) /data/mci/645997abcf6fe98d800f7fe4aac18ff9/toolchain-builder/tmp/build-llvm.sh-cv8/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:71:3 (mongod+0x22abf9d)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #1 mongo::DecorationContainer<mongo::ServiceContext>::DecorationContainer(mongo::Decorable<mongo::ServiceContext>*, mongo::DecorationRegistry<mongo::ServiceContext> const*) /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/util/decoration_container.h:100:27 (mongod+0x74915ff)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::Decorable<mongo::ServiceContext>::Decorable() /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/util/decorable.h:139:19 (mongod+0x7489a19)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::ServiceContext::ServiceContext() /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/db/service_context.cpp:101 (mongod+0x7489a19)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #4 std::_MakeUniq<mongo::ServiceContext>::__single_object std::make_unique<mongo::ServiceContext>() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/unique_ptr.h:835:34 (mongod+0x748f05e)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #5 mongo::ServiceContext::make() /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/db/service_context.cpp:509 (mongod+0x748f05e)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #6 mongo::mongod_main(int, char**)::$_67::operator()() const /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1499:41 (mongod+0x22acf79)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #7 mongo::mongod_main(int, char**) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1497 (mongod+0x22acf79)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #8 main /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/db/mongod.cpp:47:22 (mongod+0x22acc1e)
      [js_test:audit_read_from_sharded_secondaries] d21290|
      [js_test:audit_read_from_sharded_secondaries] d21290|   Thread T30 'OplogCa.plog.rs' (tid=27665, running) created by main thread at:
      [js_test:audit_read_from_sharded_secondaries] d21290|     #0 pthread_create /data/mci/645997abcf6fe98d800f7fe4aac18ff9/toolchain-builder/tmp/build-llvm.sh-cv8/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:965:3 (mongod+0x2241eb5)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (mongod+0x7939714)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::repl::StorageInterfaceImpl::initializeStorageControlsForReplication(mongo::ServiceContext*) const /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/storage_interface_impl.cpp:1418:27 (mongod+0x23ae2fc)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::repl::ReplicationCoordinatorImpl::startup(mongo::OperationContext*, mongo::StorageEngine::LastShutdownState) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/replication_coordinator_impl.cpp:932:15 (mongod+0x24c8923)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #4 mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:720:20 (mongod+0x22ba1a1)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #5 mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:863:16 (mongod+0x22b0421)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #6 mongo::mongod_main(int, char**) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1575 (mongod+0x22b0421)
      [js_test:audit_read_from_sharded_secondaries] d21290|     #7 main /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/db/mongod.cpp:47:22 (mongod+0x22acc1e)
      [js_test:audit_read_from_sharded_secondaries] d21290|
      [js_test:audit_read_from_sharded_secondaries] d21290| SUMMARY: ThreadSanitizer: data race /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/collection.h:788:34 in mongo::CollectionPtr::operator bool() const
      [js_test:audit_read_from_sharded_secondaries] d21290| ================== 
      

      Old description:

      TSAN flags some data members as being shared between threads. Made a temp PR to attempt to fix the problem but it sounds like the real issues is far more complex. See https://github.com/10gen/mongo/pull/6846/files#r958877087
      https://github.com/10gen/mongo/pull/6846/files#r961976193

      https://github.com/10gen/mongo/pull/6846/files#r961978717

       

       

      cc amirsaman.memaripour@mongodb.com 

            Assignee:
            backlog-server-repl [DO NOT USE] Backlog - Replication Team
            Reporter:
            alex.neben@mongodb.com Alex Neben
            Votes:
            0 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: