[SERVER-42871] ConnectionPool::SpecificPool is able to shutdown with outstanding requests Created: 19/Aug/19  Updated: 12/Apr/21  Resolved: 11/Oct/19

Status: Closed
Project: Core Server
Component/s: Internal Client
Affects Version/s: 4.2.0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: FirstName lipengchong Assignee: Benjamin Caimano (Inactive)
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File crash.txt    
Issue Links:
Duplicate
duplicates SERVER-42930 ConnectionPool controller updates mus... Closed
Operating System: ALL
Backport Completed:
Participants:

 Description   

We have a sharded cluster( three member replset config and 2  three member replset shards ). the  config node sometimes will crash. the BACKTRACE  blog  have been attached.



 Comments   
Comment by maneesh bhunwal [ 12/Apr/21 ]

adding 

Invariant failure pool->_requests.empty() src/mongo/executor/connection_pool.cpp 1085

for better search-ability

Comment by Benjamin Caimano (Inactive) [ 14/Oct/19 ]

Sadly, my fix caused issues on some of our more esoteric CI variants and was reverted. I'll be tracking continuing work on SERVER-42930.

Comment by Benjamin Caimano (Inactive) [ 19/Sep/19 ]

I've landed SERVER-42930 on master and v4.2. It should be released with r4.2.1.

I believe this will fix your issue. I've actually limited the invariant involved to debug mode only as well. Please reopen this ticket or submit a new one if this problem persists on r4.2.1.

Comment by Benjamin Caimano (Inactive) [ 20/Aug/19 ]

lpc, I believe this is an issue we've previously seen on our "continuous stepdown" testing suite. The most common case where this might occur is when a configsvr is relatively inactive but one of the shards it talks to is experiencing some sort of topology disruption. The logs that are attached are not sufficient to determine what was happening on the configsvr before the crash, but I suspect SERVER-42930 may solve this problem.

Comment by Danny Hatcher (Inactive) [ 19/Aug/19 ]

Symbolizer output run locally:

INFO:symbolize:build info:
{
    "edition": "community",
    "uname": {
        "release": "4.9.0-8-amd64",
        "sysname": "Linux",
        "machine": "x86_64",
        "version": "#1 SMP Debian 4.9.130-2 (2018-10-27)"
    },
    "version": "4.2.0",
    "buildId": "17838f474fa4042630d3957aabc914a7b81a067b",
    "githash": "a4b751dcf51dd249c5865812b390cfd1c0129c30"
}
INFO:symbolize:debug symbols not found in cache; downloading...
INFO:symbolize:downloading debugsymbols: https://mongosymb.s3.amazonaws.com/symbols/17838f474fa4042630d3957aabc914a7b81a067b.debug.bz2
INFO:symbolize:found symbolizer in local cache: /Users/danielhatcher/.mongosymb.cache/mongosymb.py
INFO:symbolize:detected symbolizer interpreter: python3
INFO:symbolize:running symbolizer: /usr/bin/env python3 /Users/danielhatcher/.mongosymb.cache/mongosymb.py --symbolizer-path=/Users/danielhatcher/github/triage-scripts/mongosymb/llvm-symbolizer/llvm-symbolizer-macos /Users/danielhatcher/.mongosymb.cache/17838f474fa4042630d3957aabc914a7b81a067b.debug
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/util/stacktrace_posix.cpp:174:39: mongo::printStackTrace(std::ostream&)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/util/signal_handlers_synchronous.cpp:182:20: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/util/signal_handlers_synchronous.cpp:239:28: mongo::(anonymous namespace)::abruptQuit(int)
 ??:0:0: ??
 ??:0:0: ??
 ??:0:0: ??
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/util/assert_util.cpp:104:15: mongo::invariantFailed(char const*, char const*, unsigned int)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/util/invariant.h:69:33: void mongo::invariantWithLocation<bool>(bool const&, char const*, char const*, unsigned int)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/executor/connection_pool.cpp:1085:13: mongo::executor::ConnectionPool::SpecificPool::updateController() (.cold.986)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/executor/connection_pool.cpp:1125:29: operator()
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/util/future.h:609:25: operator()
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/util/functional.h:145:10: callRegularVoid<mongo::ExecutorFuture<T>::getAsync(Func&&) && [with Func = mongo::executor::ConnectionPool::SpecificPool::updateState()::<lambda(mongo::Status&&)>; T = void]::<lambda(mongo::StatusOrStatusWith<void>)> mutable::<lambda(mongo::Status)> >
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/util/functional.h:159:39: auto mongo::unique_function<void (mongo::Status)>::makeImpl<void mongo::ExecutorFuture<void>::getAsync<mongo::executor::ConnectionPool::SpecificPool::updateState()::'lambda'(mongo::Status&&)>(mongo::executor::ConnectionPool::SpecificPool::updateState()::'lambda'(mongo::Status&&)&&) &&::'lambda'(mongo::Status)::operator()(mongo::Status)::'lambda'(mongo::Status)>(mongo::executor::ConnectionPool::SpecificPool::updateState()::'lambda'(mongo::Status&&)&&)::SpecificImpl::call(mongo::Status&&)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/util/functional.h:109:54: mongo::unique_function<void (mongo::Status)>::operator()(mongo::Status) const
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/transport/transport_layer_asio.cpp:186:63: mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()::operator()() const
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/handler_invoke_hook.hpp:68:11: void asio::asio_handler_invoke<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()>(mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()&, ...)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/detail/handler_invoke_helpers.hpp:37:22: void asio_handler_invoke_helpers::invoke<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'(), mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()>(mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()&, mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()&)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/impl/system_executor.hpp:38:38: void asio::system_executor::dispatch<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'(), std::allocator<void> >(mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()&&, std::allocator<void> const&) const
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/detail/work_dispatcher.hpp:57:5: asio::detail::work_dispatcher<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()>::operator()()
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/handler_invoke_hook.hpp:68:11: void asio::asio_handler_invoke<asio::detail::work_dispatcher<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()> >(asio::detail::work_dispatcher<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()>&, ...)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/detail/handler_invoke_helpers.hpp:37:22: void asio_handler_invoke_helpers::invoke<asio::detail::work_dispatcher<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()>, asio::detail::work_dispatcher<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()> >(asio::detail::work_dispatcher<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()>&, asio::detail::work_dispatcher<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()>&)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/detail/executor_op.hpp:69:42: asio::detail::executor_op<asio::detail::work_dispatcher<mongo::transport::TransportLayerASIO::ASIOReactor::schedule(mongo::unique_function<void (mongo::Status)>)::'lambda'()>, std::allocator<void>, asio::detail::scheduler_operation>::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/detail/scheduler_operation.hpp:39:10: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/detail/impl/scheduler.ipp:400:20: asio::detail::scheduler::do_run_one(asio::detail::conditionally_enabled_mutex::scoped_lock&, asio::detail::scheduler_thread_info&, std::error_code const&)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/detail/impl/scheduler.ipp:153:20: asio::detail::scheduler::run(std::error_code&)
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/third_party/asio-master/asio/include/asio/impl/io_context.ipp:61:27: asio::io_context::run()
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/transport/transport_layer_asio.cpp:147:27: mongo::transport::TransportLayerASIO::ASIOReactor::run()
 /data/mci/f6e74a821f5e4db4596aa072159186ae/src/src/mongo/executor/network_interface_tl.cpp:116:18: mongo::executor::NetworkInterfaceTL::_run()
 /data/mci/33dec3019633ae64bee59b3ee6bf458b/toolchain-builder/tmp/build-gcc-v3.sh-YzZ/build/x86_64-mongodb-linux/libstdc++-v3/src/c++11/../../../../../src/combined/libstdc++-v3/src/c++11/thread.cc:80:18: execute_native_thread_routine
 ??:0:0: ??
 ??:0:0: ??

Generated at Thu Feb 08 05:01:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.