[SERVER-60275] TSAN race in storage_wiredtiger_record_store_and_index_test Created: 28/Sep/21  Updated: 29/Oct/23  Resolved: 29/Sep/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 5.1.0-rc0

Type: Bug Priority: Major - P3
Reporter: Benety Goh Assignee: Benety Goh
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
is related to SERVER-59618 Avoid using LockerNoop outside of uni... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Execution Team 2021-10-04
Participants:
Linked BF Score: 60

 Description   

The following stack trace indicates there are two ServiceContext instances created by the WiredTigerHarnessHelper:

[cpp_unit_test:storage_wiredtiger_record_store_and_index_test] WARNING: ThreadSanitizer: data race (pid=45160)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]   Write of size 8 at 0x7b0400001730 by main thread:
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #0 std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> >::release() /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/unique_ptr.h:369:16 (storage_wiredtiger_record_store_and_index_test+0x5350bf2)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #1 std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> >::unique_ptr(std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> >&&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/unique_ptr.h:245 (storage_wiredtiger_record_store_and_index_test+0x5350bf2)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #2 mongo::ServiceContext::ClientObserverHolder::ClientObserverHolder(mongo::ServiceContext::ClientObserverHolder&&) /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/service_context.h:649 (storage_wiredtiger_record_store_and_index_test+0x5350bf2)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #3 void std::_Construct<mongo::ServiceContext::ClientObserverHolder, mongo::ServiceContext::ClientObserverHolder>(mongo::ServiceContext::ClientObserverHolder*, mongo::ServiceContext::ClientObserverHolder&&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_construct.h:75 (storage_wiredtiger_record_store_and_index_test+0x5350bf2)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #4 mongo::ServiceContext::ClientObserverHolder* std::__uninitialized_copy<false>::__uninit_copy<std::move_iterator<mongo::ServiceContext::ClientObserverHolder*>, mongo::ServiceContext::ClientObserverHolder*>(std::move_iterator<mongo::ServiceContext::ClientObserverHolder*>, std::move_iterator<mongo::ServiceContext::ClientObserverHolder*>, mongo::ServiceContext::ClientObserverHolder*) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_uninitialized.h:83 (storage_wiredtiger_record_store_and_index_test+0x5350bf2)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #5 mongo::ServiceContext::ClientObserverHolder* std::uninitialized_copy<std::move_iterator<mongo::ServiceContext::ClientObserverHolder*>, mongo::ServiceContext::ClientObserverHolder*>(std::move_iterator<mongo::ServiceContext::ClientObserverHolder*>, std::move_iterator<mongo::ServiceContext::ClientObserverHolder*>, mongo::ServiceContext::ClientObserverHolder*) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_uninitialized.h:131 (storage_wiredtiger_record_store_and_index_test+0x5350bf2)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #6 mongo::ServiceContext::ClientObserverHolder* std::__uninitialized_copy_a<std::move_iterator<mongo::ServiceContext::ClientObserverHolder*>, mongo::ServiceContext::ClientObserverHolder*, mongo::ServiceContext::ClientObserverHolder>(std::move_iterator<mongo::ServiceContext::ClientObserverHolder*>, std::move_iterator<mongo::ServiceContext::ClientObserverHolder*>, mongo::ServiceContext::ClientObserverHolder*, std::allocator<mongo::ServiceContext::ClientObserverHolder>&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_uninitialized.h:289 (storage_wiredtiger_record_store_and_index_test+0x5350bf2)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #7 mongo::ServiceContext::ClientObserverHolder* std::__uninitialized_move_if_noexcept_a<mongo::ServiceContext::ClientObserverHolder*, mongo::ServiceContext::ClientObserverHolder*, std::allocator<mongo::ServiceContext::ClientObserverHolder> >(mongo::ServiceContext::ClientObserverHolder*, mongo::ServiceContext::ClientObserverHolder*, mongo::ServiceContext::ClientObserverHolder*, std::allocator<mongo::ServiceContext::ClientObserverHolder>&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_uninitialized.h:310 (storage_wiredtiger_record_store_and_index_test+0x5350bf2)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #8 void std::vector<mongo::ServiceContext::ClientObserverHolder, std::allocator<mongo::ServiceContext::ClientObserverHolder> >::_M_realloc_insert<std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> > >(__gnu_cxx::__normal_iterator<mongo::ServiceContext::ClientObserverHolder*, std::vector<mongo::ServiceContext::ClientObserverHolder, std::allocator<mongo::ServiceContext::ClientObserverHolder> > >, std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> >&&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/vector.tcc:446 (storage_wiredtiger_record_store_and_index_test+0x5350bf2)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #9 mongo::ServiceContext::ClientObserverHolder& std::vector<mongo::ServiceContext::ClientObserverHolder, std::allocator<mongo::ServiceContext::ClientObserverHolder> >::emplace_back<std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> > >(std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> >&&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/vector.tcc:109:4 (storage_wiredtiger_record_store_and_index_test+0x5349461)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #10 mongo::ServiceContext::registerClientObserver(std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> >) /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/service_context.cpp:326 (storage_wiredtiger_record_store_and_index_test+0x5349461)
 
...
 
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #0 operator new(unsigned long) /data/mci/2a834e5406da55eb2b59da6bbcfb3e78/toolchain-builder/tmp/build-llvm.sh-qkG/llvm/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:65:3 (storage_wiredtiger_record_store_and_index_test+0x215fa8d)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #1 __gnu_cxx::new_allocator<mongo::ServiceContext::ClientObserverHolder>::allocate(unsigned long, void const*) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/ext/new_allocator.h:111:27 (storage_wiredtiger_record_store_and_index_test+0x5350b52)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #2 std::allocator_traits<std::allocator<mongo::ServiceContext::ClientObserverHolder> >::allocate(std::allocator<mongo::ServiceContext::ClientObserverHolder>&, unsigned long) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/alloc_traits.h:436 (storage_wiredtiger_record_store_and_index_test+0x5350b52)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #3 std::_Vector_base<mongo::ServiceContext::ClientObserverHolder, std::allocator<mongo::ServiceContext::ClientObserverHolder> >::_M_allocate(unsigned long) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/stl_vector.h:296 (storage_wiredtiger_record_store_and_index_test+0x5350b52)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #4 void std::vector<mongo::ServiceContext::ClientObserverHolder, std::allocator<mongo::ServiceContext::ClientObserverHolder> >::_M_realloc_insert<std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> > >(__gnu_cxx::__normal_iterator<mongo::ServiceContext::ClientObserverHolder*, std::vector<mongo::ServiceContext::ClientObserverHolder, std::allocator<mongo::ServiceContext::ClientObserverHolder> > >, std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> >&&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/vector.tcc:427 (storage_wiredtiger_record_store_and_index_test+0x5350b52)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #5 mongo::ServiceContext::ClientObserverHolder& std::vector<mongo::ServiceContext::ClientObserverHolder, std::allocator<mongo::ServiceContext::ClientObserverHolder> >::emplace_back<std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> > >(std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> >&&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/vector.tcc:109:4 (storage_wiredtiger_record_store_and_index_test+0x5349461)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #6 mongo::ServiceContext::registerClientObserver(std::unique_ptr<mongo::ServiceContext::ClientObserver, std::default_delete<mongo::ServiceContext::ClientObserver> >) /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/service_context.cpp:326 (storage_wiredtiger_record_store_and_index_test+0x5349461)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #7 mongo::(anonymous namespace)::$_1::operator()(mongo::ServiceContext*) const /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/auth/auth_decorations.cpp:92:18 (storage_wiredtiger_record_store_and_index_test+0x532b772)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #8 std::_Function_handler<void (mongo::ServiceContext*), mongo::(anonymous namespace)::$_1>::_M_invoke(std::_Any_data const&, mongo::ServiceContext*&&) /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/std_function.h:297 (storage_wiredtiger_record_store_and_index_test+0x532b772)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #9 std::function<void (mongo::ServiceContext*)>::operator()(mongo::ServiceContext*) const /opt/mongodbtoolchain/revisions/0d5a071f1db663c050a1d7f330c13f46e62d6d4f/stow/gcc-v3.F4j/lib/gcc/x86_64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/std_function.h:687:14 (storage_wiredtiger_record_store_and_index_test+0x534b52d)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #10 mongo::ServiceContext::ConstructorDestructorActions::onCreate(mongo::ServiceContext*) const /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/service_context.h:280 (storage_wiredtiger_record_store_and_index_test+0x534b52d)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #11 void mongo::(anonymous namespace)::onCreate<mongo::ServiceContext, std::_List_const_iterator<mongo::ServiceContext::ConstructorDestructorActions> >(mongo::ServiceContext*, std::_List_const_iterator<mongo::ServiceContext::ConstructorDestructorActions> const&, std::_List_const_iterator<mongo::ServiceContext::ConstructorDestructorActions> const&) /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/service_context.cpp:155 (storage_wiredtiger_record_store_and_index_test+0x534b52d)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #12 void mongo::(anonymous namespace)::onCreate<mongo::ServiceContext, std::__cxx11::list<mongo::ServiceContext::ConstructorDestructorActions, std::allocator<mongo::ServiceContext::ConstructorDestructorActions> > >(mongo::ServiceContext*, std::__cxx11::list<mongo::ServiceContext::ConstructorDestructorActions, std::allocator<mongo::ServiceContext::ConstructorDestructorActions> > const&) /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/service_context.cpp:165 (storage_wiredtiger_record_store_and_index_test+0x534b52d)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #13 mongo::ServiceContext::make() /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/service_context.cpp:504 (storage_wiredtiger_record_store_and_index_test+0x534b52d)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #14 mongo::ScopedGlobalServiceContextForTest::ScopedGlobalServiceContextForTest()::$_0::operator()() const /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/service_context_test_fixture.cpp:53:31 (storage_wiredtiger_record_store_and_index_test+0x252e844)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #15 mongo::ScopedGlobalServiceContextForTest::ScopedGlobalServiceContextForTest() /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/service_context_test_fixture.cpp:52 (storage_wiredtiger_record_store_and_index_test+0x252e844)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #16 mongo::HarnessHelper::HarnessHelper() /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/storage/test_harness_helper.h:57:14 (storage_wiredtiger_record_store_and_index_test+0x21abee8)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #17 mongo::RecordStoreHarnessHelper::RecordStoreHarnessHelper() /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/storage/record_store_test_harness.h:46 (storage_wiredtiger_record_store_and_index_test+0x21abee8)
[cpp_unit_test:storage_wiredtiger_record_store_and_index_test]     #18 mongo::WiredTigerHarnessHelper::WiredTigerHarnessHelper(mongo::StringData) /data/mci/1cfbd139f1796d28902bfd6cc8adbe26/src/src/mongo/db/storage/wiredtiger/wiredtiger_record_store_test_harness.cpp:38 (storage_wiredtiger_record_store_and_index_test+0x21abee8)



 Comments   
Comment by Vivian Ge (Inactive) [ 06/Oct/21 ]

Updating the fixversion since branching activities occurred yesterday. This ticket will be in rc0 when it’s been triggered. For more active release information, please keep an eye on #server-release. Thank you!

Comment by Githook User [ 29/Sep/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-60275 WiredTigerHarnessHelper registers client observer before initializing storage engine

This fixes a TSAN data race introduced in commit 6b3acc72af58963a772213cc882e34b6db5bf14.
(suppressed by partial revert in commit 505bd6a60bf925e01b4ef0b7eb501708b87c21b6).
Branch: master
https://github.com/mongodb/mongo/commit/45041b9e5aece67df50f549d67df2ff8f5fcce3e

Comment by Githook User [ 29/Sep/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-60275 add LockerNoopClientObserverRegisterer

This registration helper is intended for use in cases where the
a member field requires a ServiceContext to be updated, making it
infeasible to perform the client observer registration in the
constructor body.
Branch: master
https://github.com/mongodb/mongo/commit/4671eda7d4153bd30669b2aba85514d442f69627

Comment by Githook User [ 28/Sep/21 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-59618 SERVER-60275 temporarily remove ClientObserver registration from WiredTigerHarnessHelper
Branch: master
https://github.com/mongodb/mongo/commit/edd437e479d58cd191443c54d5af4b1b4c8287e7

Comment by Benety Goh [ 28/Sep/21 ]

This race was introduced when we added a ClientObserver in SERVER-59618.

Generated at Thu Feb 08 05:49:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.