[SERVER-68240] Iterator race condition and possible crash Created: 22/Jul/22  Updated: 29/Oct/23  Resolved: 09/Sep/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.2.0-rc0

Type: Bug Priority: Major - P3
Reporter: Alex Neben Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Service Arch 2022-09-05, Service Arch 2022-09-19
Participants:
Linked BF Score: 8

 Description   

https://spruce.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_debug_mode_audit_e37f9d5efff9974a4a66b6a84bf98f24c1052937_22_07_21_22_10_59/tests?execution=0&sortBy=STATUS&sortDir=ASC

src/mongo/db/modules/enterprise/jstests/audit/impersonated_user_metadata_edge_cases.js

 

https://logkeeper.mongodb.org/build/b2f7eb0d2c66f1ebd5844884f9ba90ee/test/62da35c9f84ae847d047de07?raw=1

 

[j0:prim] /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/debug/safe_iterator.h:165:
[j0:prim] Error: attempt to copy-construct an iterator from a singular iterator.
[j0:prim] 
[j0:prim] Objects involved in the operation:
[j0:prim]     iterator "this" @ 0x0x7f3d381781f8

{ [j0:prim]       type = __gnu_debug::_Safe_iterator<std::_Rb_tree_iterator<std::pair<mongo::repl::OpTime const, std::shared_ptr<mongo::WaitForMajorityService::Request> > >, std::__debug::multimap<mongo::repl::OpTime, std::shared_ptr<mongo::WaitForMajorityService::Request>, std::less<mongo::repl::OpTime>, std::allocator<std::pair<mongo::repl::OpTime const, std::shared_ptr<mongo::WaitForMajorityService::Request> > > > > (mutable iterator); [j0:prim]       state = singular; [j0:prim]     }

[j0:prim]     iterator "other" @ 0x0x7f3cf80bce80

{ [j0:prim]       type = __gnu_debug::_Safe_iterator<std::_Rb_tree_iterator<std::pair<mongo::repl::OpTime const, std::shared_ptr<mongo::WaitForMajorityService::Request> > >, std::__debug::multimap<mongo::repl::OpTime, std::shared_ptr<mongo::WaitForMajorityService::Request>, std::less<mongo::repl::OpTime>, std::allocator<std::pair<mongo::repl::OpTime const, std::shared_ptr<mongo::WaitForMajorityService::Request> > > > > (mutable iterator); [j0:prim]       state = singular; [j0:prim]       references sequence with type 'std::__debug::multimap<mongo::repl::OpTime, std::shared_ptr<mongo::WaitForMajorityService::Request>, std::less<mongo::repl::OpTime>, std::allocator<std::pair<mongo::repl::OpTime const, std::shared_ptr<mongo::WaitForMajorityService::Request> > > >' @ 0x0x55cc9a4aa6b8 [j0:prim]     }

[j0:prim] {"t":

{"$date":"2022-07-22T05:29:47.188+00:00"}

,"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"SignalHandler","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}



 Comments   
Comment by Githook User [ 07/Sep/22 ]

Author:

{'name': 'Blake Oler', 'email': 'blake.oler@mongodb.com', 'username': 'BlakeIsBlake'}

Message: SERVER-68240 Fix iterator race condition in WaitForMajorityService
Branch: master
https://github.com/mongodb/mongo/commit/16b22af0483333fba942c835cdc4ab5ae3dfcf23

Comment by Lingzhi Deng [ 17/Aug/22 ]

I took a quick look earlier this week. I think this is a race between WaitForMajorityService shutDown/stepDown and cancellation of majority wait requests. I think the iterator could have been invalidated due to shutDown/stepDown here (because waitForWriteConcern gets an error first) before onCancel is invoked.

Comment by Alex Neben [ 27/Jul/22 ]

 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/debug/safe_iterator.h:161:2: __gnu_debug::_Safe_iterator<std::_Rb_tree_iterator<std::pair<mongo::repl::OpTime const, std::shared_ptr<mongo::WaitForMajorityService::Request> > >, std::__debug::multimap<mongo::repl::OpTime, std::shared_ptr<mongo::WaitForMajorityService::Request>, std::less<mongo::repl::OpTime>, std::allocator<std::pair<mongo::repl::OpTime const, std::shared_ptr<mongo::WaitForMajorityService::Request> > > > >::_Safe_iterator(__gnu_debug::_Safe_iterator<std::_Rb_tree_iterator<std::pair<mongo::repl::OpTime const, std::shared_ptr<mongo::WaitForMajorityService::Request> > >, std::__debug::multimap<mongo::repl::OpTime, std::shared_ptr<mongo::WaitForMajorityService::Request>, std::less<mongo::repl::OpTime>, std::allocator<std::pair<mongo::repl::OpTime const, std::shared_ptr<mongo::WaitForMajorityService::Request> > > > >&&)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/db/repl/wait_for_majority_service.cpp:173:80: mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status)::('lambda0'(mongo::Status)&&)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/future.h:709:13: void mongo::ExecutorFuture<void>::getAsync<mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status), 0>(mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status)&&) &&::'lambda'(mongo::Status)::operator()(mongo::Status)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/future_impl.h:301:16: auto mongo::future_details::call<void mongo::ExecutorFuture<void>::getAsync<mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status), 0>(mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status)&&) &&::'lambda'(mongo::Status)&>(mongo::CleanupFuturePolicy<false>&&, mongo::StatusWith<mongo::future_details::FakeVoid>)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/future_impl.h:942:29: void mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::getAsync<mongo::CleanupFuturePolicy<false>, void mongo::ExecutorFuture<void>::getAsync<mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status), 0>(mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status)&&) &&::'lambda'(mongo::Status), 0>(mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status)&&) &&::'lambda1'()::operator()() const::'lambda'(mongo::future_details::SharedStateBase*)::operator()(mongo::future_details::SharedStateBase*)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/functional.h:259:21: auto mongo::unique_function<void (mongo::future_details::SharedStateBase*)>::makeImpl<void mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::getAsync<mongo::CleanupFuturePolicy<false>, void mongo::ExecutorFuture<void>::getAsync<mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status), 0>(mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status)&&) &&::'lambda'(mongo::Status), 0>(mongo::CleanupFuturePolicy<false>, mongo::WaitForMajorityService::waitUntilMajority(mongo::repl::OpTime const&, mongo::CancellationToken const&)::'lambda0'(mongo::Status)&&) &&::'lambda1'()::operator()() const::'lambda'(mongo::future_details::SharedStateBase*)>(mongo::CleanupFuturePolicy<false>&&)::SpecificImpl::call(mongo::future_details::SharedStateBase*&&)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/functional.h:217:54: mongo::unique_function<void (mongo::future_details::SharedStateBase*)>::operator()(mongo::future_details::SharedStateBase*) const
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/future_impl.h:482:21: mongo::future_details::SharedStateBase::transitionToFinished()
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/future_impl.h:612:29: mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>::fillFromConst(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid> const&)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/future_impl.h:656:17: mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>::fillChildren(std::__debug::forward_list<boost::intrusive_ptr<mongo::future_details::SharedStateBase>, std::allocator<boost::intrusive_ptr<mongo::future_details::SharedStateBase> > > const&) const
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/future_impl.h:497:29: mongo::future_details::SharedStateBase::transitionToFinished()
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/future_impl.h:633:29: void mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>::emplaceValue<>()
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/future.h:1267:9: void mongo::SharedPromise<void>::emplaceValue<0>()
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/cancellation.h:86:46: mongo::detail::CancellationState::cancel()
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/cancellation.h:260:36: mongo::CancellationSource::cancel()
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/db/repl/primary_only_service.cpp:437:19: mongo::repl::PrimaryOnlyService::_interruptInstances(mongo::WithLock, mongo::Status)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/db/repl/primary_only_service.cpp:467:24: mongo::repl::PrimaryOnlyService::onStepDown()
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/db/repl/primary_only_service.cpp:216:35: mongo::repl::PrimaryOnlyServiceRegistry::onStepDown()
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/db/repl/replica_set_aware_service.cpp:88:28: mongo::ReplicaSetAwareServiceRegistry::onStepDown()::'lambda'(mongo::ReplicaSetAwareInterface*)::operator()(mongo::ReplicaSetAwareInterface*) const
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/bits/stl_algo.h:3882:5: mongo::ReplicaSetAwareServiceRegistry::onStepDown()::'lambda'(mongo::ReplicaSetAwareInterface*) std::for_each<__gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<mongo::ReplicaSetAwareInterface**, std::__cxx1998::vector<mongo::ReplicaSetAwareInterface*, std::allocator<mongo::ReplicaSetAwareInterface*> > >, std::__debug::vector<mongo::ReplicaSetAwareInterface*, std::allocator<mongo::ReplicaSetAwareInterface*> > >, mongo::ReplicaSetAwareServiceRegistry::onStepDown()::'lambda'(mongo::ReplicaSetAwareInterface*)>(__gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<mongo::ReplicaSetAwareInterface**, std::__cxx1998::vector<mongo::ReplicaSetAwareInterface*, std::allocator<mongo::ReplicaSetAwareInterface*> > >, std::__debug::vector<mongo::ReplicaSetAwareInterface*, std::allocator<mongo::ReplicaSetAwareInterface*> > >, __gnu_debug::_Safe_iterator<__gnu_cxx::__normal_iterator<mongo::ReplicaSetAwareInterface**, std::__cxx1998::vector<mongo::ReplicaSetAwareInterface*, std::allocator<mongo::ReplicaSetAwareInterface*> > >, std::__debug::vector<mongo::ReplicaSetAwareInterface*, std::allocator<mongo::ReplicaSetAwareInterface*> > >, mongo::ReplicaSetAwareServiceRegistry::onStepDown()::'lambda'(mongo::ReplicaSetAwareInterface*))
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/db/repl/replica_set_aware_service.cpp:89:6: mongo::ReplicaSetAwareServiceRegistry::onStepDown()
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/db/repl/replication_coordinator_impl.cpp:4616:69: mongo::repl::ReplicationCoordinatorImpl::_performPostMemberStateUpdateAction(mongo::repl::ReplicationCoordinatorImpl::PostMemberStateUpdateAction)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp:589:40: mongo::repl::ReplicationCoordinatorImpl::_stepDownFinish(mongo::executor::TaskExecutor::CallbackArgs const&, mongo::executor::TaskExecutor::EventHandle const&)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp:516:28: mongo::repl::ReplicationCoordinatorImpl::_stepDownStart()::'lambda'(mongo::executor::TaskExecutor::CallbackArgs const&)::operator()(mongo::executor::TaskExecutor::CallbackArgs const&) const
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/functional.h:259:21: auto mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>::makeImpl<mongo::repl::ReplicationCoordinatorImpl::_stepDownStart()::'lambda'(mongo::executor::TaskExecutor::CallbackArgs const&)>(mongo::repl::ReplicationCoordinatorImpl::_stepDownStart()::'lambda'(mongo::executor::TaskExecutor::CallbackArgs const&)&&)::SpecificImpl::call(mongo::executor::TaskExecutor::CallbackArgs const&)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/functional.h:217:54: mongo::unique_function<void (mongo::executor::TaskExecutor::CallbackArgs const&)>::operator()(mongo::executor::TaskExecutor::CallbackArgs const&) const
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/executor/thread_pool_task_executor.cpp:663:17: mongo::executor::ThreadPoolTaskExecutor::runCallback(std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/executor/thread_pool_task_executor.cpp:639:28: auto mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::__debug::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, __gnu_debug::_Safe_iterator<std::__cxx1998::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >, std::__debug::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > > > const&, __gnu_debug::_Safe_iterator<std::__cxx1998::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >, std::__debug::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > > > const&, std::unique_lock<mongo::latch_detail::Latch>)::'lambda1'(auto)::operator()<mongo::Status>(auto) const
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/functional.h:259:21: auto mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::executor::ThreadPoolTaskExecutor::scheduleIntoPool_inlock(std::__debug::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > >*, __gnu_debug::_Safe_iterator<std::__cxx1998::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >, std::__debug::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > > > const&, __gnu_debug::_Safe_iterator<std::__cxx1998::_List_iterator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> >, std::__debug::list<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState>, std::allocator<std::shared_ptr<mongo::executor::ThreadPoolTaskExecutor::CallbackState> > > > const&, std::unique_lock<mongo::latch_detail::Latch>)::'lambda1'(auto)>(auto&&)::SpecificImpl::call(mongo::Status&&)
 /data/mci/a3d92df1ac6aa2525f848a2a731d255d/src/src/mongo/util/functional.h:217:54: mongo::unique_function<void (mongo::Status)>::operator()(mongo::Status) const
 /data/mci/62efd181aea92f7f19473dec0cf23227/src/src/mongo/util/concurrency/thread_pool.cpp:530:9: mongo::ThreadPool::Impl::_doOneTask(std::unique_lock<mongo::latch_detail::Latch>*)
 /data/mci/62efd181aea92f7f19473dec0cf23227/src/src/mongo/util/concurrency/thread_pool.cpp:418:23: mongo::ThreadPool::Impl::_consumeTasks()
 /data/mci/62efd181aea92f7f19473dec0cf23227/src/src/mongo/util/concurrency/thread_pool.cpp:405:18: mongo::ThreadPool::Impl::_workerThreadBody(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
 /data/mci/62efd181aea92f7f19473dec0cf23227/src/src/mongo/util/concurrency/thread_pool.cpp:580:69: mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()::operator()() const
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/bits/invoke.h:60:36: void std::__invoke_impl<void, mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()>(std::__invoke_other, mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()&&)
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/bits/invoke.h:95:40: std::__invoke_result<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()>::type std::__invoke<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()&&)
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/tuple:1678:27: decltype(auto) std::__apply_impl<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'(), std::tuple<> >(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()&&, std::tuple<>&&, std::integer_sequence<unsigned long>)
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/tuple:1687:31: decltype(auto) std::apply<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'(), std::tuple<> >(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()&&, std::tuple<>&&)
 /data/mci/62efd181aea92f7f19473dec0cf23227/src/src/mongo/stdx/thread.h:193:36: mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'(), 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'())::'lambda'()::operator()()
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/bits/invoke.h:60:36: mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'() std::__invoke_impl<void, mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'(), 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'())::'lambda'()>(std::__invoke_other, &&, 0&&...)
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/bits/invoke.h:95:40: std::__invoke_result<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()>::type std::__invoke<mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'(), 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'())::'lambda'()>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'()&&)
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/thread:244:26: decltype(__invoke(_S_declval<0ul>())) std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'(), 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'())::'lambda'()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>)
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/thread:253:31: std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'(), 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'())::'lambda'()> >::operator()()
 /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.IGP/include/c++/8.5.0/thread:196:13: std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::stdx::thread::thread<mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'(), 0>(mongo::ThreadPool::Impl::_startWorkerThread_inlock()::'lambda2'())::'lambda'()> > >::_M_run() 

Comment by Alex Neben [ 22/Jul/22 ]

Looks like also effects unit test which might make it easier to repro

https://spruce.mongodb.com/task/mongodb_mongo_master_enterprise_rhel_80_64_bit_dynamic_debug_mode_run_unittests_e37f9d5efff9974a4a66b6a84bf98f24c1052937_22_07_21_22_10_59/tests?execution=0&sortBy=STATUS&sortDir=ASC

Comment by Alex Neben [ 22/Jul/22 ]

My guess is that the iterator being captured by the lambda is getting invalidated because of a concurrent change to the map. I have spent a good 30min staring and I cannot come up with a better theory than that. This is a debug mode failures but judah.schvimer@mongodb.com if this turns our to be repl bug I want my $10 since I discovered it during skunkworks.

Generated at Thu Feb 08 06:10:16 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.