Minor - P4 After upgrading to v4.2.1 we started to get mongod processes hanging after receiving the shutdown command.
A warning ahead, we are running Fedora 30 and using the prebuilt "rhel80" binaries.
Log:
2019-11-12T23:25:53.436+0100 I NETWORK [listener] connection accepted from 127.0.0.1:37382 #3 (2 connections now open)
2019-11-12T23:25:53.436+0100 I NETWORK [conn3] received client metadata from 127.0.0.1:37382 conn3: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "4.2.1" }, os: { type: "Linux", name: "Fedora release 30 (Thirty)", architecture: "x86_64", versio n: "Kernel 5.3.8-200.fc30.x86_64" } }
2019-11-12T23:25:53.440+0100 I COMMAND [conn3] terminating, shutdown command received { shutdown: 1.0, lsid: { id: UUID("788c9cf7-ef90-44af-a3cb-bb28024fa869") }, $db: "admin" }
2019-11-12T23:25:53.440+0100 I NETWORK [conn3] shutdown: going to close listening sockets...
2019-11-12T23:25:53.440+0100 I NETWORK [conn3] removing socket file: /tmp/mongodb-23638.sock
Backtrace of mongod at this state:
(gdb) thread apply all backtrace Thread 30 (Thread 0x7fef4b911700 (LWP 63259)): #0 0x00007fef5b9c19f8 in __pthread_timedjoin_ex () from /lib64/libpthread.so.0 #1 0x00005569d17a4ae3 in std::thread::join() () #2 0x00005569d0dd8331 in mongo::transport::TransportLayerASIO::shutdown() () #3 0x00005569d0dcc839 in mongo::transport::TransportLayerManager::shutdown() () #4 0x00005569cfc28cf8 in mongo::(anonymous namespace)::shutdownTask(mongo::ShutdownTaskArgs const&) () #5 0x00005569d167a675 in mongo::(anonymous namespace)::runTasks(std::stack<mongo::unique_function<void (mongo::ShutdownTaskArgs const&)>, std::deque<mongo::unique_function<void (mongo::ShutdownTaskArgs const&)>, std::allocator<mongo::unique_function<void (mongo::ShutdownTaskArgs const&)> > > >, mongo::ShutdownTaskArgs const&) () #6 0x00005569cfba8cf7 in mongo::shutdown(mongo::ExitCode, mongo::ShutdownTaskArgs const&) () #7 0x00005569d030dbaa in mongo::CmdShutdown::shutdownHelper(mongo::BSONObj const&) () #8 0x00005569d00f2c1e in mongo::(anonymous namespace)::CmdShutdownMongoD::run(mongo::OperationContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, mongo::BSONObj const&, mongo::BSONObjBuilder&) () #9 0x00005569d108ed24 in mongo::BasicCommand::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*) () #10 0x00005569cfff2548 in mongo::(anonymous namespace)::runCommandImpl(mongo::OperationContext*, mongo::CommandInvocation*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*, mongo::LogicalTime, mongo::ServiceEntryPointCommon::Hooks const&, mongo::BSONObjBuilder*, mongo::Operation SessionInfoFromClient const&) () #11 0x00005569cfff4a34 in mongo::(anonymous namespace)::receivedCommands(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)::{lambda()#1}::operator()() const () #12 0x00005569cfff579a in mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&) () #13 0x00005569cffe3aac in mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&) () #14 0x00005569cffef94c in mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard) () #15 0x00005569cffeb16f in mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard) () #16 0x00005569cffee54c in std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::{lambda()#1 }>::_M_invoke(std::_Any_data const&) () #17 0x00005569d0dd6aa2 in mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName) () #18 0x00005569cffe8acd in mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership) () #19 0x00005569cffebe23 in mongo::ServiceStateMachine::_sourceCallback(mongo::Status) () #20 0x00005569cffea197 in mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard) () #21 0x00005569cffeb0cb in mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard) () #22 0x00005569cffee54c in std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::{lambda()#1 }>::_M_invoke(std::_Any_data const&) () #23 0x00005569d0dd6f0b in std::_Function_handler<void (), mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () #24 0x00005569d1411874 in mongo::(anonymous namespace)::runFunc(void*) () #25 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #26 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fef4ba12700 (LWP 62477)): #0 0x00007fef5b9ca8bd in recvmsg () from /lib64/libpthread.so.0 #1 0x00005569d10705b0 in asio::detail::socket_ops::recv(int, iovec*, unsigned long, int, std::error_code&) () #2 0x00005569d1070668 in asio::detail::socket_ops::sync_recv(int, unsigned char, iovec*, unsigned long, int, bool, std::error_code&) () #3 0x00005569d0df8f5b in mongo::Future<void> mongo::transport::TransportLayerASIO::ASIOSession::opportunisticRead<asio::basic_stream_socket<asio::generic::stream_protocol>, asio::mutable_buffers_1>(asio::basic_stream_socket<asio::generic::stream_protocol>&, asio::mutable_buffers_1 const&, std ::shared_ptr<mongo::Baton> const&) () #4 0x00005569d0e03eba in mongo::Future<void> mongo::transport::TransportLayerASIO::ASIOSession::read<asio::mutable_buffers_1>(asio::mutable_buffers_1 const&, std::shared_ptr<mongo::Baton> const&) () #5 0x00005569d0e075c7 in mongo::transport::TransportLayerASIO::ASIOSession::sourceMessageImpl(std::shared_ptr<mongo::Baton> const&) () #6 0x00005569d0e07a88 in mongo::transport::TransportLayerASIO::ASIOSession::sourceMessage() () #7 0x00005569cffea337 in mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard) () #8 0x00005569cffeb0cb in mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard) () #9 0x00005569cffee54c in std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::{lambda()#1 }>::_M_invoke(std::_Any_data const&) () #10 0x00005569d0dd6f0b in std::_Function_handler<void (), mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::{lambda()#1}>::_M_invoke(std::_Any_data const&) () #11 0x00005569d1411874 in mongo::(anonymous namespace)::runFunc(void*) () #12 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fef4c213700 (LWP 62168)): #0 0x00007fef5b8ee88e in epoll_wait () from /lib64/libc.so.6 #1 0x00005569d106a29e in asio::detail::epoll_reactor::run(long, asio::detail::op_queue<asio::detail::scheduler_operation>&) () #2 0x00005569d106cc7d in asio::detail::scheduler::do_run_one(asio::detail::conditionally_enabled_mutex::scoped_lock&, asio::detail::scheduler_thread_info&, std::error_code const&) () #3 0x00005569d106d205 in asio::detail::scheduler::run(std::error_code&) () #4 0x00005569d1074e5e in asio::io_context::run() () #5 0x00005569d0dd9218 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::transport::TransportLayerASIO::start()::{lambda()#1}> > >::_M_run() () #6 0x00005569d17a4a7f in execute_native_thread_routine () #7 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fef4da16700 (LWP 62165)): --Type <RET> for more, q to quit, c to continue without paging--c #0 0x00007fef5b9c973d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fef5b9c2ce9 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00005569d0dcd181 in mongo::transport::TransportLayerManager::makeBaton(mongo::OperationContext*) const () #3 0x00005569d158b011 in mongo::ServiceContext::makeOperationContext(mongo::Client*) () #4 0x00005569d1584787 in mongo::Client::makeOperationContext() () #5 0x00005569d0008120 in std::_Function_handler<void (mongo::Client*), mongo::PeriodicThreadToDecreaseSnapshotHistoryIfNotNeeded::_init(mongo::ServiceContext*)::{lambda(mongo::Client*)#1}>::_M_invoke(std::_Any_data const&, mongo::Client*&&) () #6 0x00005569d019bcb5 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1}> > >::_M_run() () #7 0x00005569d17a4a7f in execute_native_thread_routine () #8 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fef4e217700 (LWP 62164)): #0 0x00007fef5b9c973d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fef5b9c2ce9 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00005569d0dcd181 in mongo::transport::TransportLayerManager::makeBaton(mongo::OperationContext*) const () #3 0x00005569d158b011 in mongo::ServiceContext::makeOperationContext(mongo::Client*) () #4 0x00005569d1584787 in mongo::Client::makeOperationContext() () #5 0x00005569d000ae19 in std::_Function_handler<void (mongo::Client*), mongo::PeriodicThreadToAbortExpiredTransactions::_init(mongo::ServiceContext*)::{lambda(mongo::Client*)#1}>::_M_invoke(std::_Any_data const&, mongo::Client*&&) () #6 0x00005569d019bcb5 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1}> > >::_M_run() () #7 0x00005569d17a4a7f in execute_native_thread_routine () #8 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fef4ea18700 (LWP 62163)): #0 0x00007fef5b9c63c5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d17a18dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) () #2 0x00005569d0e98f6e in std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::SessionKiller::SessionKiller(mongo::ServiceContext*, std::function<mongo::StatusWith<std::vector<mongo::HostAndPort, std::allocator<mongo::HostAndPort> > > (mongo::OperationContext*, mongo::SessionKiller::Matcher const&, std::linear_congruential_engine<unsigned long, 48271ul, 0ul, 2147483647ul>*)>)::{lambda()#1}> > >::_M_run() () #3 0x00005569d17a4a7f in execute_native_thread_routine () #4 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fef4f219700 (LWP 62162)): #0 0x00007fef5b9c677a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d155cafa in mongo::(anonymous namespace)::PeriodicTaskRunner::run() () #2 0x00005569d155de1c in mongo::BackgroundJob::jobBody() () #3 0x00005569d17a4a7f in execute_native_thread_routine () #4 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fef5021b700 (LWP 62160)): #0 0x00007fef5b9c973d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fef5b9c2ce9 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00005569d0dcd181 in mongo::transport::TransportLayerManager::makeBaton(mongo::OperationContext*) const () #3 0x00005569d158b011 in mongo::ServiceContext::makeOperationContext(mongo::Client*) () #4 0x00005569d1584787 in mongo::Client::makeOperationContext() () #5 0x00005569cffdd498 in mongo::TTLMonitor::doTTLPass() () #6 0x00005569cffde1d8 in mongo::TTLMonitor::run() () #7 0x00005569d155de1c in mongo::BackgroundJob::jobBody() () #8 0x00005569d17a4a7f in execute_native_thread_routine () #9 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fef50a1c700 (LWP 62159)): #0 0x00007fef5b9c973d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fef5b9c2ce9 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00005569d0dcd181 in mongo::transport::TransportLayerManager::makeBaton(mongo::OperationContext*) const () #3 0x00005569d158b011 in mongo::ServiceContext::makeOperationContext(mongo::Client*) () #4 0x00005569d1584787 in mongo::Client::makeOperationContext() () #5 0x00005569d00b0a0b in mongo::FTDCCollectorCollection::collect(mongo::Client*) () #6 0x00005569d0094598 in mongo::FreeMonProcessor::doMetricsCollect(mongo::Client*) () #7 0x00005569d009aedb in mongo::FreeMonProcessor::run() () #8 0x00005569d17a4a7f in execute_native_thread_routine () #9 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fef5121d700 (LWP 62157)): #0 0x00007fef5b9c63c5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d17a18dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) () #2 0x00005569d04f0fad in mongo::ThreadPool::_consumeTasks() () #3 0x00005569d04f25e5 in mongo::ThreadPool::_workerThreadBody(mongo::ThreadPool*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () #4 0x00005569d17a4a7f in execute_native_thread_routine () #5 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fef51a1e700 (LWP 62156)): #0 0x00007fef5b9c63c5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d106cd9b in asio::detail::scheduler::do_run_one(asio::detail::conditionally_enabled_mutex::scoped_lock&, asio::detail::scheduler_thread_info&, std::error_code const&) () #2 0x00005569d106d205 in asio::detail::scheduler::run(std::error_code&) () #3 0x00005569d1074e5e in asio::io_context::run() () #4 0x00005569d0de2b7d in mongo::transport::TransportLayerASIO::ASIOReactor::run() () #5 0x00005569d0dc0484 in mongo::executor::NetworkInterfaceTL::_run() () #6 0x00005569d17a4a7f in execute_native_thread_routine () #7 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fef5221f700 (LWP 62155)): #0 0x00007fef5b9c973d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fef5b9c2ce9 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00005569d0dcd181 in mongo::transport::TransportLayerManager::makeBaton(mongo::OperationContext*) const () #3 0x00005569d158b011 in mongo::ServiceContext::makeOperationContext(mongo::Client*) () #4 0x00005569d1584787 in mongo::Client::makeOperationContext() () #5 0x00005569d00b0a0b in mongo::FTDCCollectorCollection::collect(mongo::Client*) () #6 0x00005569d00b4deb in mongo::FTDCController::doLoop() () #7 0x00005569d17a4a7f in execute_native_thread_routine () #8 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fef53221700 (LWP 62143)): #0 0x00007fef5b9c63c5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d106cd9b in asio::detail::scheduler::do_run_one(asio::detail::conditionally_enabled_mutex::scoped_lock&, asio::detail::scheduler_thread_info&, std::error_code const&) () #2 0x00005569d106d205 in asio::detail::scheduler::run(std::error_code&) () #3 0x00005569d1074e5e in asio::io_context::run() () #4 0x00005569d0de2b7d in mongo::transport::TransportLayerASIO::ASIOReactor::run() () #5 0x00005569d0dc0484 in mongo::executor::NetworkInterfaceTL::_run() () #6 0x00005569d17a4a7f in execute_native_thread_routine () #7 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fef53a22700 (LWP 61903)): #0 0x00007fef5b9c677a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d05fda9c in mongo::DeadlineMonitor<mongo::mozjs::MozJSImplScope>::deadlineMonitorThread() () #2 0x00005569d17a4a7f in execute_native_thread_routine () #3 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fef54223700 (LWP 61901)): #0 0x00007fef5b9c973d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fef5b9c2ce9 in pthread_mutex_lock () from /lib64/libpthread.so.0 #2 0x00005569d0dcd181 in mongo::transport::TransportLayerManager::makeBaton(mongo::OperationContext*) const () #3 0x00005569d158b011 in mongo::ServiceContext::makeOperationContext(mongo::Client*) () #4 0x00005569d1584787 in mongo::Client::makeOperationContext() () #5 0x00005569d042e22a in std::_Function_handler<void (mongo::Client*), mongo::StorageEngineImpl::TimestampMonitor::startup()::{lambda(mongo::Client*)#1}>::_M_invoke(std::_Any_data const&, mongo::Client*&&) () #6 0x00005569d019bcb5 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1}> > >::_M_run() () #7 0x00005569d17a4a7f in execute_native_thread_routine () #8 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #9 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fef54a24700 (LWP 61880)): #0 0x00007fef5b9c677a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfc7dd35 in mongo::WiredTigerKVEngine::WiredTigerCheckpointThread::run() () #2 0x00005569d155de1c in mongo::BackgroundJob::jobBody() () #3 0x00005569d17a4a7f in execute_native_thread_routine () #4 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fef55225700 (LWP 61879)): #0 0x00007fef5b9ca0a5 in nanosleep () from /lib64/libpthread.so.0 #1 0x00005569d1685a0e in mongo::sleepmillis(long long) () #2 0x00005569cfc7d7ec in mongo::WiredTigerKVEngine::WiredTigerJournalFlusher::run() () #3 0x00005569d155de1c in mongo::BackgroundJob::jobBody() () #4 0x00005569d17a4a7f in execute_native_thread_routine () #5 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fef55a26700 (LWP 61878)): #0 0x00007fef5b9c677a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfc7d378 in mongo::WiredTigerKVEngine::WiredTigerSessionSweeper::run() () #2 0x00005569d155de1c in mongo::BackgroundJob::jobBody() () #3 0x00005569d17a4a7f in execute_native_thread_routine () #4 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fef56227700 (LWP 61877)): #0 0x00007fef5b9c672c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfcde55b in __wt_cond_wait_signal () #2 0x00005569cfcba36c in __sweep_server () #3 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fef56a28700 (LWP 61876)): #0 0x00007fef5b9c672c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfcde55b in __wt_cond_wait_signal () #2 0x00005569cfd08a46 in __wt_cond_auto_wait_signal () #3 0x00005569cfd08a93 in __wt_cond_auto_wait () #4 0x00005569cfcc9f7c in __wt_evict_thread_run () #5 0x00005569cfd167e9 in __thread_run () #6 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fef57229700 (LWP 61875)): #0 0x00007fef5b9c672c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfcde55b in __wt_cond_wait_signal () #2 0x00005569cfd08a46 in __wt_cond_auto_wait_signal () #3 0x00005569cfd08a93 in __wt_cond_auto_wait () #4 0x00005569cfcc9f7c in __wt_evict_thread_run () #5 0x00005569cfd167e9 in __thread_run () #6 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fef57a2a700 (LWP 61874)): #0 0x00007fef5b9c672c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfcde55b in __wt_cond_wait_signal () #2 0x00005569cfd08a46 in __wt_cond_auto_wait_signal () #3 0x00005569cfd08a93 in __wt_cond_auto_wait () #4 0x00005569cfcc9f7c in __wt_evict_thread_run () #5 0x00005569cfd167e9 in __thread_run () #6 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fef5822b700 (LWP 61873)): #0 0x00007fef5b9c672c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfcde55b in __wt_cond_wait_signal () #2 0x00005569cfd08a46 in __wt_cond_auto_wait_signal () #3 0x00005569cfd08a93 in __wt_cond_auto_wait () #4 0x00005569cfcc9f7c in __wt_evict_thread_run () #5 0x00005569cfd167e9 in __thread_run () #6 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fef58a2c700 (LWP 61856)): #0 0x00007fef5b9c672c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfcde55b in __wt_cond_wait_signal () #2 0x00005569cfd08a46 in __wt_cond_auto_wait_signal () #3 0x00005569cfd8d94a in __log_server () #4 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fef5922d700 (LWP 61855)): #0 0x00007fef5b9c672c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfcde55b in __wt_cond_wait_signal () #2 0x00005569cfd08a46 in __wt_cond_auto_wait_signal () #3 0x00005569cfd08a93 in __wt_cond_auto_wait () #4 0x00005569cfd8ea93 in __log_wrlsn_server () #5 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fef59a2e700 (LWP 61854)): #0 0x00007fef5b9c672c in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569cfcde55b in __wt_cond_wait_signal () #2 0x00005569cfd8dc2b in __log_file_server () #3 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fef5a22f700 (LWP 61464)): #0 0x00007fef5b9c677a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d158e817 in mongo::ClockSource::waitForConditionUntil(mongo::stdx::condition_variable&, std::unique_lock<std::mutex>&, mongo::Date_t, mongo::Waitable*) () #2 0x00005569d019bf28 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1}> > >::_M_run() () #3 0x00005569d17a4a7f in execute_native_thread_routine () #4 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fef5aa30700 (LWP 61463)): #0 0x00007fef5b9c63c5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d17a18dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) () #2 0x00005569d158dd97 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<mongo::BackgroundThreadClockSource::_startTimerThread()::{lambda()#1}> > >::_M_run() () #3 0x00005569d17a4a7f in execute_native_thread_routine () #4 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fef5b231700 (LWP 61462)): #0 0x00007fef5b9c63c5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d17a18dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) () #2 0x00005569cfba8a4c in mongo::shutdown(mongo::ExitCode, mongo::ShutdownTaskArgs const&) () #3 0x00005569d031c903 in mongo::(anonymous namespace)::signalProcessingThread(mongo::LogFileStatus) () #4 0x00005569d17a4a7f in execute_native_thread_routine () #5 0x00007fef5b9c04c0 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fef5b8ee553 in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fef5b232c00 (LWP 61455)): #0 0x00007fef5b9c63c5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00005569d17a18dc in std::condition_variable::wait(std::unique_lock<std::mutex>&) () #2 0x00005569d167a76f in mongo::waitForShutdown() () #3 0x00005569cfc2b786 in mongo::(anonymous namespace)::_initAndListen(int) [clone .isra.508] () #4 0x00005569cfc2cf5d in mongo::(anonymous namespace)::mongoDbMain(int, char**, char**) () #5 0x00005569cfbb25b9 in main ()