[SERVER-69419] Race condition in setting/accessing LocalOplogInfo::_oplog CollectionPtr Created: 02/Sep/22  Updated: 05/Dec/22  Resolved: 10/Nov/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Alex Neben Assignee: Backlog - Replication Team
Resolution: Duplicate Votes: 0
Labels: repl-shortlist, tsan
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Duplicate
duplicates SERVER-70551 Fix data race between acquireOplogCol... Closed
Assigned Teams:
Replication
Operating System: ALL
Participants:
Linked BF Score: 24

 Description   

The race condition is around acquireOplogCollectionForLogging() attempting to set LocalOplogInfo::_oplog at the same time OplogCapMaintainerThread::_deleteExcessDocuments() is attempting to access it. Forwarding to repl.

[js_test:audit_read_from_sharded_secondaries] d21290| ==================
[js_test:audit_read_from_sharded_secondaries] d21290| WARNING: ThreadSanitizer: data race (pid=27563)
[js_test:audit_read_from_sharded_secondaries] d21290|   Read of size 8 at 0x7ba80003b0e0 by thread T30:
[js_test:audit_read_from_sharded_secondaries] d21290|     #0 mongo::CollectionPtr::operator bool() const /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/collection.h:788:34 (mongod+0x23c70c6)
[js_test:audit_read_from_sharded_secondaries] d21290|     #1 mongo::OplogCapMaintainerThread::_deleteExcessDocuments() /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/storage/oplog_cap_maintainer_thread.cpp:75 (mongod+0x23c70c6)
[js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::OplogCapMaintainerThread::run() /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/storage/oplog_cap_maintainer_thread.cpp:114:14 (mongod+0x23c7e8d)
[js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::BackgroundJob::jobBody() /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/util/background.cpp:162:5 (mongod+0x73fd08a)
[js_test:audit_read_from_sharded_secondaries] d21290|     #4 mongo::BackgroundJob::go()::$_0::operator()() const /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/util/background.cpp:188:31 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #5 void std::__invoke_impl<void, mongo::BackgroundJob::go()::$_0>(std::__invoke_other, mongo::BackgroundJob::go()::$_0&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:60 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #6 std::__invoke_result<mongo::BackgroundJob::go()::$_0>::type std::__invoke<mongo::BackgroundJob::go()::$_0>(mongo::BackgroundJob::go()::$_0&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:95 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #7 decltype(auto) std::__apply_impl<mongo::BackgroundJob::go()::$_0, std::tuple<> >(mongo::BackgroundJob::go()::$_0&&, std::tuple<>&&, std::integer_sequence<unsigned long>) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1678 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #8 decltype(auto) std::apply<mongo::BackgroundJob::go()::$_0, std::tuple<> >(mongo::BackgroundJob::go()::$_0&&, std::tuple<>&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1687 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #9 mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()::operator()() /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/stdx/thread.h:194 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #10 mongo::BackgroundJob::go()::$_0 std::__invoke_impl<void, mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()>(std::__invoke_other, mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:60 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #11 std::__invoke_result<mongo::BackgroundJob::go()::$_0>::type std::__invoke<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()>(mongo::BackgroundJob::go()::$_0&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:95 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #12 decltype(std::__invoke(_S_declval<0ul>())) std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/thread:244 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #13 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()> >::operator()() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/thread:253 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()> > >::_M_run() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/thread:196 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #15 execute_native_thread_routine <null> (mongod+0x793948e)
[js_test:audit_read_from_sharded_secondaries] d21290|
[js_test:audit_read_from_sharded_secondaries] d21290|   Previous write of size 8 at 0x7ba80003b0e0 by main thread:
[js_test:audit_read_from_sharded_secondaries] d21290|     #0 mongo::CollectionPtr::operator=(mongo::CollectionPtr&&) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/collection.cpp:52:56 (mongod+0x701cfb2)
[js_test:audit_read_from_sharded_secondaries] d21290|     #1 mongo::LocalOplogInfo::setCollection(mongo::CollectionPtr const&) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/local_oplog_info.cpp:71:12 (mongod+0x48c8dec)
[js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::repl::acquireOplogCollectionForLogging(mongo::OperationContext*) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/oplog.cpp:2081:33 (mongod+0x3f3ae3a)
[js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::repl::ReplicationCoordinatorImpl::startup(mongo::OperationContext*, mongo::StorageEngine::LastShutdownState) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/replication_coordinator_impl.cpp:947:9 (mongod+0x24c89c2)
[js_test:audit_read_from_sharded_secondaries] d21290|     #4 mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:720:20 (mongod+0x22ba1a1)
[js_test:audit_read_from_sharded_secondaries] d21290|     #5 mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:863:16 (mongod+0x22b0421)
[js_test:audit_read_from_sharded_secondaries] d21290|     #6 mongo::mongod_main(int, char**) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1575 (mongod+0x22b0421)
[js_test:audit_read_from_sharded_secondaries] d21290|     #7 main /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/db/mongod.cpp:47:22 (mongod+0x22acc1e)
[js_test:audit_read_from_sharded_secondaries] d21290|
[js_test:audit_read_from_sharded_secondaries] d21290|   Location is heap block of size 23921 at 0x7ba800036000 allocated by main thread:
[js_test:audit_read_from_sharded_secondaries] d21290|     #0 operator new[](unsigned long) /data/mci/645997abcf6fe98d800f7fe4aac18ff9/toolchain-builder/tmp/build-llvm.sh-cv8/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:71:3 (mongod+0x22abf9d)
[js_test:audit_read_from_sharded_secondaries] d21290|     #1 mongo::DecorationContainer<mongo::ServiceContext>::DecorationContainer(mongo::Decorable<mongo::ServiceContext>*, mongo::DecorationRegistry<mongo::ServiceContext> const*) /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/util/decoration_container.h:100:27 (mongod+0x74915ff)
[js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::Decorable<mongo::ServiceContext>::Decorable() /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/util/decorable.h:139:19 (mongod+0x7489a19)
[js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::ServiceContext::ServiceContext() /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/db/service_context.cpp:101 (mongod+0x7489a19)
[js_test:audit_read_from_sharded_secondaries] d21290|     #4 std::_MakeUniq<mongo::ServiceContext>::__single_object std::make_unique<mongo::ServiceContext>() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/unique_ptr.h:835:34 (mongod+0x748f05e)
[js_test:audit_read_from_sharded_secondaries] d21290|     #5 mongo::ServiceContext::make() /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/db/service_context.cpp:509 (mongod+0x748f05e)
[js_test:audit_read_from_sharded_secondaries] d21290|     #6 mongo::mongod_main(int, char**)::$_67::operator()() const /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1499:41 (mongod+0x22acf79)
[js_test:audit_read_from_sharded_secondaries] d21290|     #7 mongo::mongod_main(int, char**) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1497 (mongod+0x22acf79)
[js_test:audit_read_from_sharded_secondaries] d21290|     #8 main /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/db/mongod.cpp:47:22 (mongod+0x22acc1e)
[js_test:audit_read_from_sharded_secondaries] d21290|
[js_test:audit_read_from_sharded_secondaries] d21290|   Thread T30 'OplogCa.plog.rs' (tid=27665, running) created by main thread at:
[js_test:audit_read_from_sharded_secondaries] d21290|     #0 pthread_create /data/mci/645997abcf6fe98d800f7fe4aac18ff9/toolchain-builder/tmp/build-llvm.sh-cv8/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:965:3 (mongod+0x2241eb5)
[js_test:audit_read_from_sharded_secondaries] d21290|     #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (mongod+0x7939714)
[js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::repl::StorageInterfaceImpl::initializeStorageControlsForReplication(mongo::ServiceContext*) const /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/storage_interface_impl.cpp:1418:27 (mongod+0x23ae2fc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::repl::ReplicationCoordinatorImpl::startup(mongo::OperationContext*, mongo::StorageEngine::LastShutdownState) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/replication_coordinator_impl.cpp:932:15 (mongod+0x24c8923)
[js_test:audit_read_from_sharded_secondaries] d21290|     #4 mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:720:20 (mongod+0x22ba1a1)
[js_test:audit_read_from_sharded_secondaries] d21290|     #5 mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:863:16 (mongod+0x22b0421)
[js_test:audit_read_from_sharded_secondaries] d21290|     #6 mongo::mongod_main(int, char**) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1575 (mongod+0x22b0421)
[js_test:audit_read_from_sharded_secondaries] d21290|     #7 main /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/db/mongod.cpp:47:22 (mongod+0x22acc1e)
[js_test:audit_read_from_sharded_secondaries] d21290|
[js_test:audit_read_from_sharded_secondaries] d21290| SUMMARY: ThreadSanitizer: data race /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/collection.h:788:34 in mongo::CollectionPtr::operator bool() const
[js_test:audit_read_from_sharded_secondaries] d21290| ================== 

Old description:

TSAN flags some data members as being shared between threads. Made a temp PR to attempt to fix the problem but it sounds like the real issues is far more complex. See https://github.com/10gen/mongo/pull/6846/files#r958877087
https://github.com/10gen/mongo/pull/6846/files#r961976193

https://github.com/10gen/mongo/pull/6846/files#r961978717

 

 

cc amirsaman.memaripour@mongodb.com 



 Comments   
Comment by Lingzhi Deng [ 10/Nov/22 ]

I think this is a dup of SERVER-70551. SERVER-70551 should fix the issue by initializing the oplog collection pointer earlier before starting any storage control threads (e.g. OplogCapMaintainerThread). Closing as dup

Comment by Jason Chan [ 12/Sep/22 ]

Investigate OpCtx usage, can consider sending to other teams once SA is able to give guidance on how it should be used.

Comment by Alex Neben [ 06/Sep/22 ]

Note to self, once this ticket is completed add a lot more tests to TSAN.

Comment by Alex Neben [ 06/Sep/22 ]

https://spruce.mongodb.com/task/mongodb_mongo_master_enterprise_rhel80_debug_tsan_audit_patch_f04904e58c6d63fd8cc9171f43291564583c486f_63178912850e611d03efebe7_22_09_06_17_54_14/tests?execution=0&sortBy=STATUS&sortDir=ASC

Comment by Alex Neben [ 06/Sep/22 ]

[js_test:audit_read_from_sharded_secondaries] d21290| ==================
[js_test:audit_read_from_sharded_secondaries] d21290| WARNING: ThreadSanitizer: data race (pid=27563)
[js_test:audit_read_from_sharded_secondaries] d21290|   Read of size 8 at 0x7ba80003b0e0 by thread T30:
[js_test:audit_read_from_sharded_secondaries] d21290|     #0 mongo::CollectionPtr::operator bool() const /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/collection.h:788:34 (mongod+0x23c70c6)
[js_test:audit_read_from_sharded_secondaries] d21290|     #1 mongo::OplogCapMaintainerThread::_deleteExcessDocuments() /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/storage/oplog_cap_maintainer_thread.cpp:75 (mongod+0x23c70c6)
[js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::OplogCapMaintainerThread::run() /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/storage/oplog_cap_maintainer_thread.cpp:114:14 (mongod+0x23c7e8d)
[js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::BackgroundJob::jobBody() /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/util/background.cpp:162:5 (mongod+0x73fd08a)
[js_test:audit_read_from_sharded_secondaries] d21290|     #4 mongo::BackgroundJob::go()::$_0::operator()() const /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/util/background.cpp:188:31 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #5 void std::__invoke_impl<void, mongo::BackgroundJob::go()::$_0>(std::__invoke_other, mongo::BackgroundJob::go()::$_0&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:60 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #6 std::__invoke_result<mongo::BackgroundJob::go()::$_0>::type std::__invoke<mongo::BackgroundJob::go()::$_0>(mongo::BackgroundJob::go()::$_0&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:95 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #7 decltype(auto) std::__apply_impl<mongo::BackgroundJob::go()::$_0, std::tuple<> >(mongo::BackgroundJob::go()::$_0&&, std::tuple<>&&, std::integer_sequence<unsigned long>) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1678 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #8 decltype(auto) std::apply<mongo::BackgroundJob::go()::$_0, std::tuple<> >(mongo::BackgroundJob::go()::$_0&&, std::tuple<>&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/tuple:1687 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #9 mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()::operator()() /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/stdx/thread.h:194 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #10 mongo::BackgroundJob::go()::$_0 std::__invoke_impl<void, mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()>(std::__invoke_other, mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:60 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #11 std::__invoke_result<mongo::BackgroundJob::go()::$_0>::type std::__invoke<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()>(mongo::BackgroundJob::go()::$_0&&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/invoke.h:95 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #12 decltype(std::__invoke(_S_declval<0ul>())) std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/thread:244 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #13 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()> >::operator()() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/thread:253 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #14 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::BackgroundJob::go()::$_0, 0>(mongo::BackgroundJob::go()::$_0)::'lambda'()> > >::_M_run() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/thread:196 (mongod+0x73ff8dc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #15 execute_native_thread_routine <null> (mongod+0x793948e)
[js_test:audit_read_from_sharded_secondaries] d21290|
[js_test:audit_read_from_sharded_secondaries] d21290|   Previous write of size 8 at 0x7ba80003b0e0 by main thread:
[js_test:audit_read_from_sharded_secondaries] d21290|     #0 mongo::CollectionPtr::operator=(mongo::CollectionPtr&&) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/collection.cpp:52:56 (mongod+0x701cfb2)
[js_test:audit_read_from_sharded_secondaries] d21290|     #1 mongo::LocalOplogInfo::setCollection(mongo::CollectionPtr const&) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/local_oplog_info.cpp:71:12 (mongod+0x48c8dec)
[js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::repl::acquireOplogCollectionForLogging(mongo::OperationContext*) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/oplog.cpp:2081:33 (mongod+0x3f3ae3a)
[js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::repl::ReplicationCoordinatorImpl::startup(mongo::OperationContext*, mongo::StorageEngine::LastShutdownState) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/replication_coordinator_impl.cpp:947:9 (mongod+0x24c89c2)
[js_test:audit_read_from_sharded_secondaries] d21290|     #4 mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:720:20 (mongod+0x22ba1a1)
[js_test:audit_read_from_sharded_secondaries] d21290|     #5 mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:863:16 (mongod+0x22b0421)
[js_test:audit_read_from_sharded_secondaries] d21290|     #6 mongo::mongod_main(int, char**) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1575 (mongod+0x22b0421)
[js_test:audit_read_from_sharded_secondaries] d21290|     #7 main /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/db/mongod.cpp:47:22 (mongod+0x22acc1e)
[js_test:audit_read_from_sharded_secondaries] d21290|
[js_test:audit_read_from_sharded_secondaries] d21290|   Location is heap block of size 23921 at 0x7ba800036000 allocated by main thread:
[js_test:audit_read_from_sharded_secondaries] d21290|     #0 operator new[](unsigned long) /data/mci/645997abcf6fe98d800f7fe4aac18ff9/toolchain-builder/tmp/build-llvm.sh-cv8/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:71:3 (mongod+0x22abf9d)
[js_test:audit_read_from_sharded_secondaries] d21290|     #1 mongo::DecorationContainer<mongo::ServiceContext>::DecorationContainer(mongo::Decorable<mongo::ServiceContext>*, mongo::DecorationRegistry<mongo::ServiceContext> const*) /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/util/decoration_container.h:100:27 (mongod+0x74915ff)
[js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::Decorable<mongo::ServiceContext>::Decorable() /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/util/decorable.h:139:19 (mongod+0x7489a19)
[js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::ServiceContext::ServiceContext() /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/db/service_context.cpp:101 (mongod+0x7489a19)
[js_test:audit_read_from_sharded_secondaries] d21290|     #4 std::_MakeUniq<mongo::ServiceContext>::__single_object std::make_unique<mongo::ServiceContext>() /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/unique_ptr.h:835:34 (mongod+0x748f05e)
[js_test:audit_read_from_sharded_secondaries] d21290|     #5 mongo::ServiceContext::make() /data/mci/d158a84fbf39734a4cce637369fe3419/src/src/mongo/db/service_context.cpp:509 (mongod+0x748f05e)
[js_test:audit_read_from_sharded_secondaries] d21290|     #6 mongo::mongod_main(int, char**)::$_67::operator()() const /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1499:41 (mongod+0x22acf79)
[js_test:audit_read_from_sharded_secondaries] d21290|     #7 mongo::mongod_main(int, char**) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1497 (mongod+0x22acf79)
[js_test:audit_read_from_sharded_secondaries] d21290|     #8 main /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/db/mongod.cpp:47:22 (mongod+0x22acc1e)
[js_test:audit_read_from_sharded_secondaries] d21290|
[js_test:audit_read_from_sharded_secondaries] d21290|   Thread T30 'OplogCa.plog.rs' (tid=27665, running) created by main thread at:
[js_test:audit_read_from_sharded_secondaries] d21290|     #0 pthread_create /data/mci/645997abcf6fe98d800f7fe4aac18ff9/toolchain-builder/tmp/build-llvm.sh-cv8/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:965:3 (mongod+0x2241eb5)
[js_test:audit_read_from_sharded_secondaries] d21290|     #1 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) <null> (mongod+0x7939714)
[js_test:audit_read_from_sharded_secondaries] d21290|     #2 mongo::repl::StorageInterfaceImpl::initializeStorageControlsForReplication(mongo::ServiceContext*) const /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/storage_interface_impl.cpp:1418:27 (mongod+0x23ae2fc)
[js_test:audit_read_from_sharded_secondaries] d21290|     #3 mongo::repl::ReplicationCoordinatorImpl::startup(mongo::OperationContext*, mongo::StorageEngine::LastShutdownState) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/repl/replication_coordinator_impl.cpp:932:15 (mongod+0x24c8923)
[js_test:audit_read_from_sharded_secondaries] d21290|     #4 mongo::(anonymous namespace)::_initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:720:20 (mongod+0x22ba1a1)
[js_test:audit_read_from_sharded_secondaries] d21290|     #5 mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:863:16 (mongod+0x22b0421)
[js_test:audit_read_from_sharded_secondaries] d21290|     #6 mongo::mongod_main(int, char**) /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/mongod_main.cpp:1575 (mongod+0x22b0421)
[js_test:audit_read_from_sharded_secondaries] d21290|     #7 main /data/mci/38b1c0f15357e467662c94140d5e7fab/src/src/mongo/db/mongod.cpp:47:22 (mongod+0x22acc1e)
[js_test:audit_read_from_sharded_secondaries] d21290|
[js_test:audit_read_from_sharded_secondaries] d21290| SUMMARY: ThreadSanitizer: data race /data/mci/07de919e3721ab824858f71a73c58065/src/src/mongo/db/catalog/collection.h:788:34 in mongo::CollectionPtr::operator bool() const
[js_test:audit_read_from_sharded_secondaries] d21290| ================== 

Comment by Alex Neben [ 06/Sep/22 ]

https://logkeeper2.build.10gen.cc/build/181fa2224a97bffc92748d727cbac81d/test/17125a2b98f22d62da7f09c0f3e30c5c?raw=1

Comment by Jason Chan [ 06/Sep/22 ]

alexander.neben@mongodb.com would you mind linking the BFs that are flagging the races?

Comment by Billy Donahue [ 05/Sep/22 ]

link to logs?

Generated at Thu Feb 08 06:13:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.