[SERVER-49030] Fix unsynchronized uses of CurOp member variables Created: 23/Jun/20  Updated: 29/Oct/23  Resolved: 04/Aug/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 4.7.0

Type: Bug Priority: Major - P3
Reporter: Gregory Noma Assignee: Justin Seyster
Resolution: Fixed Votes: 0
Labels: qexec-team, thread-sanitizer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:
Linked BF Score: 0

 Description   

After SERVER-48970, TSAN reports unsynchronized uses of _start in CurOp here and here. It's possible that there are more private members of CurOp with similar synchronization problems that are hidden behind the current failure on _start described, so I've named this ticket to cover any subsequent failures in CurOp as well.



 Comments   
Comment by Githook User [ 04/Aug/20 ]

Author:

{'name': 'Justin Seyster', 'email': 'justin.seyster@mongodb.com', 'username': 'jseyster'}

Message: SERVER-49030 Make CurOp::_start and CurOp::_end atomic
Branch: master
https://github.com/mongodb/mongo/commit/b579403c660583fdf657ad379da235537fe2396b

Comment by Ian Boros [ 29/Jul/20 ]

Interesting. I don't know about everyone else, but the fact that _end is written multiple times seems a little suspicious. The way I see it, an "ideal" situation would be that the thread executing under the operation context associated with the CurrentOp sets '_end' once (and as you suggest, is the only thread ever to write '_end'). I don't know if this is realistic though.

The invariant definitely seems like a good sanity check to have. I'd be curious where it is violated (especially the relaxed version which allows opCtx to be nullptr).

Comment by Andrew Morrow (Inactive) [ 29/Jun/20 ]

david.storch - Great, thanks for the clarification.

Comment by David Storch [ 29/Jun/20 ]

acm regarding your comment about this ticket not being in a sprint – that is just a quirk of our team's scheduling process. We are devoting our Friday build failure time to fixing this. So it is indeed scheduled to be fixed.

Comment by Eric Milkie [ 29/Jun/20 ]

There is no _size member in the CurOp class – did you mean _start?

Comment by Andrew Morrow (Inactive) [ 27/Jun/20 ]

justin.seyster - Thanks for analyzing the possibilities. My preference would be to start with option 2, which avoids any C++ UB even if the underlying reality of the hardware would render this race "benign". If implementing option 2 proves to have unacceptably adverse performance impact, I think I would be OK with option 1. I don't have any particular opinion on option 3 as I'm unfamiliar with the details of the code, but it seems plausible to me that the extra locking is something we would prefer to avoid.

Comment by Justin Seyster [ 26/Jun/20 ]

This is a straightforward and (depending on our assumptions about the platform) benign data race. The '_start' variable in a CurOp is initialized to zero and will get written at most one time by CurOp::ensureStarted(), which sets '_start' to the current time iff it still has its initial zero value.

A $currentOp (or similar) operation can read CurOp objects associated with other threads, including reading their '_start' values. $currentOp knows to ignore the value if it is still 0 (i.e., ensureStarted() has not yet been called on the thread that owns the CurOp), but it's still a data race.

As far as I know, all the readers of CurOp::_size hold the client lock while reading, but most callers to CurOp::ensureStarted() do not hold the lock. I'm aware of three options.

1) We could annotate CurOp::ensureStarted() with _attribute_((no_sanitize_thread)), which allows this test to pass with TSan enabled (because TSan never sees any writes to CurOp::_size). Even though there is a data race, there is no bug, so long as we assume that there can be no torn reads/writes of the CurOp::_size variable (which is a 64-bit int). I don't know if there is a server-wide policy about whether we consider 64-bit loads and stores to be atomic.

2) We could give CurOp::_size the std::atomic() treatment, eliminating the possibility of undefined behavior. I haven't tried it, but I think that would be good enough to satisfy TSan.

3) We could require that all callers of CurOp::ensureStarted() hold the client lock. Even though it's only one extra lock acquisition per operation, it's still probably not worth it.

Comment by Justin Seyster [ 26/Jun/20 ]

For convenience, here is the the TSan output that this ticket is intended to address:

[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.133+0000 ==================
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000 WARNING: ThreadSanitizer: data race (pid=28049)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000   Read of size 8 at 0x7fff14cf4698 by thread T28 (mutexes: write M985579476991660072, write M268944031667458088):
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #0 mongo::CurOp::reportState(mongo::OperationContext*, mongo::BSONObjBuilder*, bool) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/curop.cpp:610:9 (mongo_embedded_test+0x1c9e5dc)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #1 mongo::CurOp::reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, bool, bool, mongo::BSONObjBuilder*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/curop.cpp:318:34 (mongo_embedded_test+0x1c9de05)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #2 mongo::CommonMongodProcessInterface::_reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/pipeline/process_interface/common_mongod_process_interface.cpp:460:5 (mongo_embedded_test+0x108750c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #3 mongo::CommonProcessInterface::getCurrentOps(boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::MongoProcessInterface::CurrentOpConnectionsMode, mongo::MongoProcessInterface::CurrentOpSessionsMode, mongo::MongoProcessInterface::CurrentOpUserMode, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpCursorMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/pipeline/process_interface/common_process_interface.cpp:97:26 (mongo_embedded_test+0x10a00fd)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #4 mongo::DocumentSourceCurrentOp::doGetNext() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/pipeline/document_source_current_op.cpp:113:48 (mongo_embedded_test+0x18f077e)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #5 mongo::DocumentSource::getNext() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/pipeline/document_source.h:259:20 (mongo_embedded_test+0x18afb9c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #6 mongo::DocumentSourceGroup::initialize() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/pipeline/document_source_group.cpp:483:36 (mongo_embedded_test+0x192b522)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #7 mongo::DocumentSourceGroup::doGetNext() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/pipeline/document_source_group.cpp:136:43 (mongo_embedded_test+0x192b371)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #8 mongo::DocumentSource::getNext() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/pipeline/document_source.h:259:20 (mongo_embedded_test+0x18afb9c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #9 mongo::Pipeline::getNext() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/pipeline/pipeline.cpp:454:40 (mongo_embedded_test+0x1a06c90)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #10 mongo::PipelineProxyStage::getNext() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/exec/pipeline_proxy.cpp:128:23 (mongo_embedded_test+0x16c1780)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #11 mongo::PipelineProxyStage::doWork(unsigned long*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/exec/pipeline_proxy.cpp:82:21 (mongo_embedded_test+0x16c0f42)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #12 mongo::PlanStage::work(unsigned long*) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/db/exec/plan_stage.h:205:26 (mongo_embedded_test+0x167742e)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #13 mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::Document>*, mongo::RecordId*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/query/plan_executor_impl.cpp:536:45 (mongo_embedded_test+0x174cbef)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #14 mongo::PlanExecutorImpl::getNext(mongo::Document*, mongo::RecordId*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/query/plan_executor_impl.cpp:394:23 (mongo_embedded_test+0x174c48a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #15 mongo::(anonymous namespace)::handleCursorCommand(mongo::OperationContext*, boost::intrusive_ptr<mongo::ExpressionContext>, mongo::NamespaceString const&, std::vector<mongo::ClientCursor*, std::allocator<mongo::ClientCursor*> >, mongo::AggregationRequest const&, mongo::rpc::ReplyBuilderInterface*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/commands/run_aggregate.cpp:175:27 (mongo_embedded_test+0x12b9305)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #16 mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::LiteParsedPipeline const&, mongo::BSONObj const&, std::vector<mongo::Privilege, std::allocator<mongo::Privilege> > const&, mongo::rpc::ReplyBuilderInterface*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/commands/run_aggregate.cpp:758 (mongo_embedded_test+0x12b9305)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.134+0000     #17 mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, std::vector<mongo::Privilege, std::allocator<mongo::Privilege> > const&, mongo::rpc::ReplyBuilderInterface*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/commands/run_aggregate.cpp:485:12 (mongo_embedded_test+0x12b521c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #18 mongo::CurrentOpCommand::runAggregation(mongo::OperationContext*, mongo::AggregationRequest const&) const /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/commands/current_op.cpp:78:23 (mongo_embedded_test+0x1257e11)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #19 mongo::CurrentOpCommandBase::run(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::BSONObjBuilder&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/commands/current_op_common.cpp:117:23 (mongo_embedded_test+0x132b7ec)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #20 mongo::BasicCommand::runWithReplyBuilder(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/commands.h:807:16 (mongo_embedded_test+0xd87bfc)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #21 mongo::BasicCommandWithReplyBuilderInterface::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/commands.cpp:770:29 (mongo_embedded_test+0x1ccdc7d)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #22 mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/commands.cpp:184:17 (mongo_embedded_test+0x1cbf09e)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #23 mongo::(anonymous namespace)::runCommandImpl(mongo::OperationContext*, mongo::CommandInvocation*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::LogicalTime, mongo::ServiceEntryPointCommon::Hooks const&, mongo::BSONObjBuilder*, mongo::OperationSessionInfoFromClient const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/service_entry_point_common.cpp:835:13 (mongo_embedded_test+0x1162a82)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #24 mongo::(anonymous namespace)::execCommandDatabase(mongo::OperationContext*, mongo::Command*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::ServiceEntryPointCommon::Hooks const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/service_entry_point_common.cpp:1176 (mongo_embedded_test+0x1162a82)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #25 mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)::$_14::operator()() const /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/service_entry_point_common.cpp:1348:13 (mongo_embedded_test+0x1159790)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #26 mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/service_entry_point_common.cpp:1275 (mongo_embedded_test+0x1159790)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #27 mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/service_entry_point_common.cpp:1682 (mongo_embedded_test+0x1159790)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #28 mongo::ServiceEntryPointEmbedded::handleRequest(mongo::OperationContext*, mongo::Message const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/embedded/service_entry_point_embedded.cpp:119:12 (mongo_embedded_test+0xda8b98)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #29 mongo::(anonymous namespace)::client_wire_protocol_rpc(mongo_embedded_v1_client*, void const*, unsigned long, void**, unsigned long*) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:429:29 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #30 mongo_embedded_v1_client_invoke::$_6::operator()() const /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:517 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #31 int mongo::enterCXXImpl<mongo::StatusForAPI<mongo_embedded_v1_error>, mongo_embedded_v1_client_invoke::$_6, void>::call<mongo_embedded_v1_client_invoke::$_6>(mongo_embedded_v1_client_invoke::$_6&&, mongo::StatusForAPI<mongo_embedded_v1_error>&, mongo::ReentrancyGuard const&&) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/api_common.h:156 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #32 decltype(mongo::enterCXXImpl<mongo::StatusForAPI<mongo_embedded_v1_error>, mongo_embedded_v1_client_invoke::$_6>::call(std::forward<mongo_embedded_v1_client_invoke::$_6>(fp0), *(fp))) mongo::enterCXX<mongo::StatusForAPI<mongo_embedded_v1_error>, mongo_embedded_v1_client_invoke::$_6>(mongo::StatusForAPI<mongo_embedded_v1_error>*, mongo_embedded_v1_client_invoke::$_6&&) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/api_common.h:203 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #33 mongo_embedded_v1_client_invoke /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:516 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #34 (anonymous namespace)::MongodbCAPITest::performRpc(std::unique_ptr<mongo_embedded_v1_client, (anonymous namespace)::ClientDestructor>&, mongo::OpMsgRequest) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/embedded/mongo_embedded/mongo_embedded_test.cpp:179:19 (mongo_embedded_test+0xd01255)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.135+0000     #35 (anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0::operator()() const /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/embedded/mongo_embedded/mongo_embedded_test.cpp:359:26 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #36 void std::__invoke_impl<void, (anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0>(std::__invoke_other, (anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0&&) /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/invoke.h:60 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #37 std::__invoke_result<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0>::type std::__invoke<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0>((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0&&) /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/invoke.h:95 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #38 decltype(auto) std::__apply_impl<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0, std::tuple<> >((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0&&, std::tuple<>&&, std::integer_sequence<unsigned long>) /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/tuple:1678 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #39 decltype(auto) std::apply<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0, std::tuple<> >((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0&&, std::tuple<>&&) /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/tuple:1687 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #40 mongo::stdx::thread::thread<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0, 0>((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0)::'lambda'()::operator()() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/stdx/thread.h:186 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #41 (anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0 std::__invoke_impl<void, mongo::stdx::thread::thread<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0, 0>((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0)::'lambda'()>(std::__invoke_other, mongo::stdx::thread::thread<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0, 0>((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0)::'lambda'()&&) /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/invoke.h:60 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #42 std::__invoke_result<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0>::type std::__invoke<mongo::stdx::thread::thread<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0, 0>((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0)::'lambda'()>((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0&&) /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/invoke.h:95 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #43 decltype(std::__invoke(_S_declval<0ul>())) std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0, 0>((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0)::'lambda'()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/thread:234 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #44 std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0, 0>((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0)::'lambda'()> >::operator()() /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/thread:243 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #45 std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0, 0>((anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest()::$_0)::'lambda'()> > >::_M_run() /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/thread:186 (mongo_embedded_test+0xd06b3c)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #46 execute_native_thread_routine /data/mci/853b8ee2d7c2b0391a18edccb3c6d9e5/toolchain-builder/tmp/build-gcc-v3.sh-mma/build/x86_64-mongodb-linux/libstdc++-v3/src/c++11/../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80:18 (mongo_embedded_test+0x3de5a0e)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000 
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000   Previous write of size 8 at 0x7fff14cf4698 by main thread (mutexes: write M1135886925576638256):
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.136+0000     #0 mongo::CurOp::ensureStarted() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/curop.cpp:414:16 (mongo_embedded_test+0x1ca0d34)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #1 mongo::(anonymous namespace)::execCommandDatabase(mongo::OperationContext*, mongo::Command*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::ServiceEntryPointCommon::Hooks const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/service_entry_point_common.cpp:1157:28 (mongo_embedded_test+0x11624ab)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #2 mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)::$_14::operator()() const /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/service_entry_point_common.cpp:1348:13 (mongo_embedded_test+0x1159790)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #3 mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/service_entry_point_common.cpp:1275 (mongo_embedded_test+0x1159790)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #4 mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/service_entry_point_common.cpp:1682 (mongo_embedded_test+0x1159790)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #5 mongo::ServiceEntryPointEmbedded::handleRequest(mongo::OperationContext*, mongo::Message const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/embedded/service_entry_point_embedded.cpp:119:12 (mongo_embedded_test+0xda8b98)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #6 mongo::(anonymous namespace)::loopbackBuildResponse(mongo::OperationContext*, mongo::LastError*, mongo::Message&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/dbdirectclient.cpp:146:64 (mongo_embedded_test+0x1840fe0)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #7 mongo::DBDirectClient::call(mongo::Message&, mongo::Message&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/db/dbdirectclient.cpp:151:23 (mongo_embedded_test+0x1840d31)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #8 mongo::DBClientBase::runCommandWithTarget(mongo::OpMsgRequest) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/client/dbclient_base.cpp:221:5 (mongo_embedded_test+0x330f665)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #9 mongo::DBClientBase::runCommandWithTarget(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj, mongo::BSONObj&, int) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/client/dbclient_base.cpp:252:19 (mongo_embedded_test+0x3310af2)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #10 mongo::DBClientBase::runCommand(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj, mongo::BSONObj&, int) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/client/dbclient_base.cpp:272:16 (mongo_embedded_test+0x330de15)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #11 mongo::SessionsCollectionStandalone::setupSessionsCollection(mongo::OperationContext*) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/db/sessions_collection_standalone.cpp:63:21 (mongo_embedded_test+0x11547db)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #12 mongo::LogicalSessionCacheImpl::_refresh(mongo::Client*) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/db/logical_session_cache_impl.cpp:266:24 (mongo_embedded_test+0x1347e4b)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #13 mongo::LogicalSessionCacheImpl::_periodicRefresh(mongo::Client*) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/db/logical_session_cache_impl.cpp:127:9 (mongo_embedded_test+0x1349e44)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.137+0000     #14 mongo::LogicalSessionCacheImpl::LogicalSessionCacheImpl(std::unique_ptr<mongo::ServiceLiaison, std::default_delete<mongo::ServiceLiaison> >, std::shared_ptr<mongo::SessionsCollection>, mongo::unique_function<int (mongo::OperationContext*, mongo::SessionsCollection&, mongo::Date_t)>)::$_12::operator()(mongo::Client*) const /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/db/logical_session_cache_impl.cpp:72:57 (mongo_embedded_test+0x134b603)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #15 std::_Function_handler<void (mongo::Client*), mongo::LogicalSessionCacheImpl::LogicalSessionCacheImpl(std::unique_ptr<mongo::ServiceLiaison, std::default_delete<mongo::ServiceLiaison> >, std::shared_ptr<mongo::SessionsCollection>, mongo::unique_function<int (mongo::OperationContext*, mongo::SessionsCollection&, mongo::Date_t)>)::$_12>::_M_invoke(std::_Any_data const&, mongo::Client*&&) /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/std_function.h:297 (mongo_embedded_test+0x134b603)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #16 std::function<void (mongo::Client*)>::operator()(mongo::Client*) const /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/std_function.h:687:14 (mongo_embedded_test+0xd8df37)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #17 mongo::PeriodicRunnerEmbedded::tryPump() /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/periodic_runner_embedded.cpp:129 (mongo_embedded_test+0xd8df37)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #18 mongo::ServiceEntryPointEmbedded::handleRequest(mongo::OperationContext*, mongo::Message const&) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/embedded/service_entry_point_embedded.cpp:118:11 (mongo_embedded_test+0xda8b7b)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #19 mongo::(anonymous namespace)::client_wire_protocol_rpc(mongo_embedded_v1_client*, void const*, unsigned long, void**, unsigned long*) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:429:29 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #20 mongo_embedded_v1_client_invoke::$_6::operator()() const /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:517 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #21 int mongo::enterCXXImpl<mongo::StatusForAPI<mongo_embedded_v1_error>, mongo_embedded_v1_client_invoke::$_6, void>::call<mongo_embedded_v1_client_invoke::$_6>(mongo_embedded_v1_client_invoke::$_6&&, mongo::StatusForAPI<mongo_embedded_v1_error>&, mongo::ReentrancyGuard const&&) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/api_common.h:156 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #22 decltype(mongo::enterCXXImpl<mongo::StatusForAPI<mongo_embedded_v1_error>, mongo_embedded_v1_client_invoke::$_6>::call(std::forward<mongo_embedded_v1_client_invoke::$_6>(fp0), *(fp))) mongo::enterCXX<mongo::StatusForAPI<mongo_embedded_v1_error>, mongo_embedded_v1_client_invoke::$_6>(mongo::StatusForAPI<mongo_embedded_v1_error>*, mongo_embedded_v1_client_invoke::$_6&&) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/api_common.h:203 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #23 mongo_embedded_v1_client_invoke /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/embedded/mongo_embedded/mongo_embedded.cpp:516 (mongo_embedded_test+0xd6784a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #24 (anonymous namespace)::MongodbCAPITest::performRpc(std::unique_ptr<mongo_embedded_v1_client, (anonymous namespace)::ClientDestructor>&, mongo::OpMsgRequest) /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/embedded/mongo_embedded/mongo_embedded_test.cpp:179:19 (mongo_embedded_test+0xd01255)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #25 (anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp::_doTest() /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/embedded/mongo_embedded/mongo_embedded_test.cpp:386:23 (mongo_embedded_test+0xd05e12)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #26 mongo::unittest::Test::run() /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/unittest/unittest.cpp:241:9 (mongo_embedded_test+0xd411e4)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #27 mongo::unittest::Test::RegistrationAgent<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp>::RegistrationAgent(mongo::StringData, mongo::StringData, mongo::StringData)::'lambda'()::operator()() const /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/unittest/unittest.h:529:21 (mongo_embedded_test+0xd05abd)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #28 std::_Function_handler<void (), mongo::unittest::Test::RegistrationAgent<(anonymous namespace)::UnitTest_SuiteNameMongodbCAPITestTestNameKillOp>::RegistrationAgent(mongo::StringData, mongo::StringData, mongo::StringData)::'lambda'()>::_M_invoke(std::_Any_data const&) /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/std_function.h:297 (mongo_embedded_test+0xd05abd)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #29 std::function<void ()>::operator()() const /opt/mongodbtoolchain/revisions/e5348beb43e147b74a40f4ca5fb05a330ea646cf/stow/gcc-v3.sKi/lib/gcc/x86_64-mongodb-linux/8.2.0/../../../../include/c++/8.2.0/bits/std_function.h:687:14 (mongo_embedded_test+0xd43c97)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.138+0000     #30 mongo::unittest::Suite::run(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/unittest/unittest.cpp:423 (mongo_embedded_test+0xd43c97)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.139+0000     #31 mongo::unittest::Suite::run(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) /data/mci/a5a68b81091e38a2a0e479995b3ec177/src/src/mongo/unittest/unittest.cpp:492:26 (mongo_embedded_test+0xd45d8a)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.139+0000     #32 main /data/mci/9dc0ba330ec7d4ad52739e2c92a99aa4/src/src/mongo/embedded/mongo_embedded/mongo_embedded_test.cpp:770:25 (mongo_embedded_test+0xcfdfae)
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.139+0000 
[cpp_unit_test:mongo_embedded_test] 2020-06-18T21:18:35.139+0000   Location is stack of main thread.

Comment by Andrew Morrow (Inactive) [ 25/Jun/20 ]

I notice that this ticket got scheduled to "backlog" and not queued into a sprint. Without fixing this, we will be unable to get the unit test suite to fully green on the TSAN builder, which will block expanding testing to include dbtest, and subsequently the broader suite of real integration tests. Would it be possible to get this scheduled into a future sprint?

Generated at Thu Feb 08 05:18:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.