[SERVER-47541] mongod segfaults during mongot e2e tests Created: 14/Apr/20  Updated: 27/Apr/20  Resolved: 27/Apr/20

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Kevin Rosendahl Assignee: Benjamin Caimano (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongod-logs.txt     Text File segfault-1.txt     Text File segfault-2.txt    
Issue Links:
Duplicate
duplicates SERVER-45261 create more state in NiTL ctor Closed
Related
is related to SERVER-45261 create more state in NiTL ctor Closed
Operating System: ALL
Sprint: Service arch 2020-05-04
Participants:

 Description   

While running our e2e tests for mongot

I'll attach the segfault and full mongod logs. Generally what's happening in these tests is some very small amount of data being loaded into different collections, mongot will collection scan and open a change stream on the collections, and a $searchBeta aggregation will be run against the collection from our test runner. These tests are run against a 4.2.3 mongod.



 Comments   
Comment by Kevin Rosendahl [ 24/Apr/20 ]

kevin.rosendahl, I'd like to confirm, is "mongod-logs.txt" a log for a single mongod?

Yes, this was a mongod being run in a docker container, it was restarted each time it crashed but with the same arguments (data directory, etc).

I think this might actually be fixed by SERVER-45261 which was released in r4.2.4 (approximately a month ago). kevin.rosendahl, are you able to upgrade a patch version?

Sure, we have a ticket to upgrade these tests to 4.2.5 (CLOUDP-57337). I'm not sure I'll be able to confidently say whether or not that definitely fixes things, as these exceptions are pretty flakey, but if you're decently confident that that may be the issue we can close this out and I can re-open if it resurfaces.

Comment by Benjamin Caimano (Inactive) [ 24/Apr/20 ]

I think this might actually be fixed by SERVER-45261 which was released in r4.2.4 (approximately a month ago). kevin.rosendahl, are you able to upgrade a patch version?

Comment by Benjamin Caimano (Inactive) [ 24/Apr/20 ]

Thanks for the backtraces, carl.champain!

It looks like we're segfaulting inside a relatively innocuous spot in the connection pool infra. There is a very conspicuous "No TransportLayer configured during NetworkInterface startup" shown here. That usually means that we're attempting to use a "substitute" networking layer before official startup happens. It might be a red herring but it is something for me to pursue.

kevin.rosendahl, I'd like to confirm, is "mongod-logs.txt" a log for a single mongod?

Comment by Carl Champain (Inactive) [ 15/Apr/20 ]

Second symbolized backtrace:

INFO:symbolize:build info:
{
    "uname": {
        "sysname": "Linux",
        "release": "4.15.0-1044-aws",
        "version": "#46-Ubuntu SMP Thu Jul 4 13:38:28 UTC 2019",
        "machine": "x86_64"
    },
    "version": "4.2.3",
    "githash": "6874650b362138df74be53d366bbefc321ea32d4",
    "edition": "enterprise",
    "buildId": "e107d3d5776183762f12f38e7afe1fe50e5ccb58"
}
INFO:symbolize:found symbols file in local cache: /Users/kelsey/.mongosymb.cache/e107d3d5776183762f12f38e7afe1fe50e5ccb58.debug
INFO:symbolize:found symbolizer in local cache: /Users/kelsey/.mongosymb.cache/mongosymb.py
INFO:symbolize:detected symbolizer interpreter: python3
INFO:symbolize:running symbolizer: /usr/bin/env python3 /Users/kelsey/.mongosymb.cache/mongosymb.py --symbolizer-path=/Users/kelsey/repos/triage-scripts/mongosymb/llvm-symbolizer/llvm-symbolizer-macos /Users/kelsey/.mongosymb.cache/e107d3d5776183762f12f38e7afe1fe50e5ccb58.debug
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/stacktrace_posix.cpp:174:39: mongo::printStackTrace(std::ostream&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/signal_handlers_synchronous.cpp:182:20: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/signal_handlers_synchronous.cpp:281:28: mongo::(anonymous namespace)::abruptQuitWithAddrSignal(int, siginfo_t*, void*)
 ??:0:0: ??
 ??:0:0: ??
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/x86_64-mongodb-linux/bits/gthr-default.h:757:44: __gthread_mutex_trylock
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_mutex.h:114:38: std::mutex::try_lock()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/platform/mutex.cpp:35:24: mongo::Mutex::lock()
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_mutex.h:162:9: _ZNSt10lock_guardIN5mongo5MutexEEC4ERS1_
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/connection_pool.cpp:531:31: mongo::executor::ConnectionPool::get(mongo::HostAndPort const&, mongo::transport::ConnectSSLMode, mongo::Duration<std::ratio<1l, 1000l> >)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/network_interface_tl.cpp:330:91: mongo::executor::NetworkInterfaceTL::startCommand(mongo::executor::TaskExecutor::CallbackHandle const&, mongo::executor::RemoteCommandRequestImpl<std::vector<mongo::HostAndPort, std::allocator<mongo::HostAndPort> > >&, mongo::unique_function<void (mongo::executor::RemoteCommandOnAnyResponse const&)>&&, std::shared_ptr<mongo::Baton> const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/thread_pool_task_executor.cpp:483:14: mongo::executor::ThreadPoolTaskExecutor::scheduleRemoteCommandOnAny(mongo::executor::RemoteCommandRequestImpl<std::vector<mongo::HostAndPort, std::allocator<mongo::HostAndPort> > > const&, std::function<void (mongo::executor::TaskExecutor::RemoteCommandOnAnyCallbackArgs const&)> const&, std::shared_ptr<mongo::Baton> const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/task_executor.cpp:120:6: mongo::executor::TaskExecutor::scheduleRemoteCommand(mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort> const&, std::function<void (mongo::executor::TaskExecutor::RemoteCommandCallbackArgs const&)> const&, std::shared_ptr<mongo::Baton> const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/task_executor_cursor.cpp:116:17: mongo::executor::TaskExecutorCursor::_runRemoteCommand(mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort> const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/task_executor_cursor.cpp:55:22: mongo::executor::TaskExecutorCursor::TaskExecutorCursor(mongo::executor::TaskExecutor*, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort> const&, mongo::executor::TaskExecutorCursor::Options&&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/third_party/boost-1.70.0/boost/optional/optional.hpp:421:7: void boost::optional_detail::optional_base<mongo::executor::TaskExecutorCursor>::construct<mongo::executor::TaskExecutor*&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&>(boost::optional_ns::in_place_init_t, mongo::executor::TaskExecutor*&&&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&&&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/third_party/boost-1.70.0/boost/optional/optional.hpp:429:7: void boost::optional_detail::optional_base<mongo::executor::TaskExecutorCursor>::emplace_assign<mongo::executor::TaskExecutor*&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&>(mongo::executor::TaskExecutor*&&&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&&&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/third_party/boost-1.70.0/boost/optional/optional.hpp:1106:7: void boost::optional<mongo::executor::TaskExecutorCursor>::emplace<mongo::executor::TaskExecutor*&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&>(mongo::executor::TaskExecutor*&&&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&&&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/modules/enterprise/src/search_beta/document_source_internal_search_beta_mongot_remote.cpp:59:20: mongo::DocumentSourceInternalSearchBetaMongotRemote::populateCursor()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/modules/enterprise/src/search_beta/document_source_internal_search_beta_mongot_remote.cpp:91:23: mongo::DocumentSourceInternalSearchBetaMongotRemote::getNext()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/modules/enterprise/src/search_beta/document_source_internal_search_beta_id_lookup.cpp:53:43: mongo::DocumentSourceInternalSearchBetaIdLookUp::getNext()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/pipeline/document_source_single_document_transformation.cpp:64:35: mongo::DocumentSourceSingleDocumentTransformation::getNext()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/pipeline/pipeline.cpp:473:48: mongo::Pipeline::getNext()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/exec/pipeline_proxy.cpp:125:40: mongo::PipelineProxyStage::getNextBson()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/exec/pipeline_proxy.cpp:82:53: mongo::PipelineProxyStage::doWork(unsigned long*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/exec/plan_stage.cpp:47:35: mongo::PlanStage::work(unsigned long*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/query/plan_executor_impl.cpp:528:49: mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*, mongo::RecordId*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/query/plan_executor_impl.cpp:381:35: mongo::PlanExecutorImpl::getNext(mongo::BSONObj*, mongo::RecordId*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/commands/run_aggregate.cpp:183:34: mongo::(anonymous namespace)::handleCursorCommand(mongo::OperationContext*, mongo::NamespaceString const&, std::vector<mongo::ClientCursor*, std::allocator<mongo::ClientCursor*> >, mongo::AggregationRequest const&, mongo::rpc::ReplyBuilderInterface*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/commands/run_aggregate.cpp:779:32: mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, std::vector<mongo::Privilege, std::allocator<mongo::Privilege> > const&, mongo::rpc::ReplyBuilderInterface*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/commands/pipeline_command.cpp:109:13: mongo::(anonymous namespace)::PipelineCommand::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:560:32: mongo::(anonymous namespace)::runCommandImpl(mongo::OperationContext*, mongo::CommandInvocation*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::LogicalTime, mongo::ServiceEntryPointCommon::Hooks const&, mongo::BSONObjBuilder*, mongo::OperationSessionInfoFromClient const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:888:32: execCommandDatabase
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:1029:32: mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)::'lambda'()::operator()() const
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:1044:6: receivedCommands
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:1329:38: mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_mongod.cpp:262:68: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:441:72: mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:519:32: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:561:29: operator()
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::'lambda'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_executor_synchronous.cpp:107:17: mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:564:80: mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:362:38: mongo::ServiceStateMachine::_sourceCallback(mongo::Status)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:318:24: operator()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future_impl.h:236:16: call<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>&, mongo::StatusWith<mongo::Message> >
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future_impl.h:855:32: operator()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future_impl.h:1170:27: generalImpl<mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Message&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Status&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda()> >
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future_impl.h:869:14: getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> >
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future.h:349:9: getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> >
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:313:29: mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:516:31: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:561:29: operator()
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::'lambda'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_executor_synchronous.cpp:124:36: operator()
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::'lambda'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_entry_point_utils.cpp:56:15: mongo::(anonymous namespace)::runFunc(void*)
 ??:0:0: ??
 ??:0:0: ??

 

Comment by Carl Champain (Inactive) [ 15/Apr/20 ]

First symbolized backtrace:

INFO:symbolize:build info:
{
    "uname": {
        "sysname": "Linux",
        "release": "4.15.0-1044-aws",
        "version": "#46-Ubuntu SMP Thu Jul 4 13:38:28 UTC 2019",
        "machine": "x86_64"
    },
    "version": "4.2.3",
    "githash": "6874650b362138df74be53d366bbefc321ea32d4",
    "edition": "enterprise",
    "buildId": "e107d3d5776183762f12f38e7afe1fe50e5ccb58"
}
INFO:symbolize:found symbols file in local cache: /Users/kelsey/.mongosymb.cache/e107d3d5776183762f12f38e7afe1fe50e5ccb58.debug
INFO:symbolize:found symbolizer in local cache: /Users/kelsey/.mongosymb.cache/mongosymb.py
INFO:symbolize:detected symbolizer interpreter: python3
INFO:symbolize:running symbolizer: /usr/bin/env python3 /Users/kelsey/.mongosymb.cache/mongosymb.py --symbolizer-path=/Users/kelsey/repos/triage-scripts/mongosymb/llvm-symbolizer/llvm-symbolizer-macos /Users/kelsey/.mongosymb.cache/e107d3d5776183762f12f38e7afe1fe50e5ccb58.debug
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/stacktrace_posix.cpp:174:39: mongo::printStackTrace(std::ostream&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/signal_handlers_synchronous.cpp:182:20: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/signal_handlers_synchronous.cpp:281:28: mongo::(anonymous namespace)::abruptQuitWithAddrSignal(int, siginfo_t*, void*)
 ??:0:0: ??
 ??:0:0: ??
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/x86_64-mongodb-linux/bits/gthr-default.h:757:44: __gthread_mutex_trylock
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_mutex.h:114:38: std::mutex::try_lock()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/platform/mutex.cpp:35:24: mongo::Mutex::lock()
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_mutex.h:162:9: _ZNSt10lock_guardIN5mongo5MutexEEC4ERS1_
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/connection_pool.cpp:531:31: mongo::executor::ConnectionPool::get(mongo::HostAndPort const&, mongo::transport::ConnectSSLMode, mongo::Duration<std::ratio<1l, 1000l> >)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/network_interface_tl.cpp:330:91: mongo::executor::NetworkInterfaceTL::startCommand(mongo::executor::TaskExecutor::CallbackHandle const&, mongo::executor::RemoteCommandRequestImpl<std::vector<mongo::HostAndPort, std::allocator<mongo::HostAndPort> > >&, mongo::unique_function<void (mongo::executor::RemoteCommandOnAnyResponse const&)>&&, std::shared_ptr<mongo::Baton> const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/thread_pool_task_executor.cpp:483:14: mongo::executor::ThreadPoolTaskExecutor::scheduleRemoteCommandOnAny(mongo::executor::RemoteCommandRequestImpl<std::vector<mongo::HostAndPort, std::allocator<mongo::HostAndPort> > > const&, std::function<void (mongo::executor::TaskExecutor::RemoteCommandOnAnyCallbackArgs const&)> const&, std::shared_ptr<mongo::Baton> const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/task_executor.cpp:120:6: mongo::executor::TaskExecutor::scheduleRemoteCommand(mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort> const&, std::function<void (mongo::executor::TaskExecutor::RemoteCommandCallbackArgs const&)> const&, std::shared_ptr<mongo::Baton> const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/task_executor_cursor.cpp:116:17: mongo::executor::TaskExecutorCursor::_runRemoteCommand(mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort> const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/executor/task_executor_cursor.cpp:55:22: mongo::executor::TaskExecutorCursor::TaskExecutorCursor(mongo::executor::TaskExecutor*, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort> const&, mongo::executor::TaskExecutorCursor::Options&&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/third_party/boost-1.70.0/boost/optional/optional.hpp:421:7: void boost::optional_detail::optional_base<mongo::executor::TaskExecutorCursor>::construct<mongo::executor::TaskExecutor*&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&>(boost::optional_ns::in_place_init_t, mongo::executor::TaskExecutor*&&&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&&&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/third_party/boost-1.70.0/boost/optional/optional.hpp:429:7: void boost::optional_detail::optional_base<mongo::executor::TaskExecutorCursor>::emplace_assign<mongo::executor::TaskExecutor*&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&>(mongo::executor::TaskExecutor*&&&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&&&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/third_party/boost-1.70.0/boost/optional/optional.hpp:1106:7: void boost::optional<mongo::executor::TaskExecutorCursor>::emplace<mongo::executor::TaskExecutor*&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&>(mongo::executor::TaskExecutor*&&&, mongo::executor::RemoteCommandRequestImpl<mongo::HostAndPort>&&&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/modules/enterprise/src/search_beta/document_source_internal_search_beta_mongot_remote.cpp:59:20: mongo::DocumentSourceInternalSearchBetaMongotRemote::populateCursor()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/modules/enterprise/src/search_beta/document_source_internal_search_beta_mongot_remote.cpp:91:23: mongo::DocumentSourceInternalSearchBetaMongotRemote::getNext()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/modules/enterprise/src/search_beta/document_source_internal_search_beta_id_lookup.cpp:53:43: mongo::DocumentSourceInternalSearchBetaIdLookUp::getNext()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/pipeline/document_source_single_document_transformation.cpp:64:35: mongo::DocumentSourceSingleDocumentTransformation::getNext()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/pipeline/pipeline.cpp:473:48: mongo::Pipeline::getNext()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/exec/pipeline_proxy.cpp:125:40: mongo::PipelineProxyStage::getNextBson()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/exec/pipeline_proxy.cpp:82:53: mongo::PipelineProxyStage::doWork(unsigned long*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/exec/plan_stage.cpp:47:35: mongo::PlanStage::work(unsigned long*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/query/plan_executor_impl.cpp:528:49: mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*, mongo::RecordId*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/query/plan_executor_impl.cpp:381:35: mongo::PlanExecutorImpl::getNext(mongo::BSONObj*, mongo::RecordId*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/commands/run_aggregate.cpp:183:34: mongo::(anonymous namespace)::handleCursorCommand(mongo::OperationContext*, mongo::NamespaceString const&, std::vector<mongo::ClientCursor*, std::allocator<mongo::ClientCursor*> >, mongo::AggregationRequest const&, mongo::rpc::ReplyBuilderInterface*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/commands/run_aggregate.cpp:779:32: mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, std::vector<mongo::Privilege, std::allocator<mongo::Privilege> > const&, mongo::rpc::ReplyBuilderInterface*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/commands/pipeline_command.cpp:109:13: mongo::(anonymous namespace)::PipelineCommand::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:560:32: mongo::(anonymous namespace)::runCommandImpl(mongo::OperationContext*, mongo::CommandInvocation*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::LogicalTime, mongo::ServiceEntryPointCommon::Hooks const&, mongo::BSONObjBuilder*, mongo::OperationSessionInfoFromClient const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:888:32: execCommandDatabase
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:1029:32: mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)::'lambda'()::operator()() const
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:1044:6: receivedCommands
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_common.cpp:1329:38: mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/db/service_entry_point_mongod.cpp:262:68: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:441:72: mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:519:32: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:561:29: operator()
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::'lambda'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_executor_synchronous.cpp:107:17: mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:564:80: mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:362:38: mongo::ServiceStateMachine::_sourceCallback(mongo::Status)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:318:24: operator()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future_impl.h:236:16: call<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>&, mongo::StatusWith<mongo::Message> >
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future_impl.h:855:32: operator()
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future_impl.h:1170:27: generalImpl<mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Message&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Status&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda()> >
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future_impl.h:869:14: getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> >
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/util/future.h:349:9: getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> >
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:313:29: mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:516:31: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_state_machine.cpp:561:29: operator()
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::'lambda'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_executor_synchronous.cpp:124:36: operator()
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:297:37: std::_Function_handler<void (), mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::'lambda'()>::_M_invoke(std::_Any_data const&)
 /opt/mongodbtoolchain/revisions/23805b43dd027076b4ae48533ab385e20c61e0cf/stow/gcc-v3.U14/include/c++/8.2.0/bits/std_function.h:687:14: std::function<void ()>::operator()() const
 /data/mci/c1288f3405f2649ccba27e463256b3d1/src/src/mongo/transport/service_entry_point_utils.cpp:56:15: mongo::(anonymous namespace)::runFunc(void*)
 ??:0:0: ??
 ??:0:0: ??

Generated at Thu Feb 08 05:14:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.