Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-37404

Terminate called on AsyncRequestsSender destructor

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 4.1.4
    • Affects Version/s: None
    • Component/s: Sharding
    • Labels:
      None
    • Fully Compatible
    • ALL
    • Sharding 2018-10-08

      In logical session cache sharding refresh suites, we occasionally see terminate called from the AsyncRequestsSender on shutdown.

      [ShardedClusterFixture:job3:configsvr:primary] 2018-09-30T20:43:46.127+0000 F -        [LogicalSessionCacheRefresh] terminate() called. An exception is active; attempting to gather more information
      [ShardedClusterFixture:job3:configsvr:primary] 2018-09-30T20:43:46.142+0000 F -        [LogicalSessionCacheRefresh] DBException::toString(): InterruptedAtShutdown: interrupted at shutdown
      [ShardedClusterFixture:job3:configsvr:primary] Actual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)11600, mongo::ExceptionForCat<(mongo::ErrorCategory)1>, mongo::ExceptionForCat<(mongo::ErrorCategory)6> >
      

      Lobster

      Symbolized backtrace:

      /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/util/stacktrace_posix.cpp:172:0: mongo::printStackTrace(std::ostream&)
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/util/signal_handlers_synchronous.cpp:229:0: mongo::(anonymous namespace)::myTerminate()
       /data/mci/05f0e86cbdc92f18b9a81c80a01536dd/toolchain-builder/build-gcc-v2.sh-Zsd/ppc64le-mongodb-linux/libstdc++-v3/libsupc++/../../../../gcc-5.4.0/libstdc++-v3/libsupc++/eh_terminate.cc:47:0: __cxxabiv1::__terminate(void (*)())
       /data/mci/05f0e86cbdc92f18b9a81c80a01536dd/toolchain-builder/build-gcc-v2.sh-Zsd/ppc64le-mongodb-linux/libstdc++-v3/libsupc++/../../../../gcc-5.4.0/libstdc++-v3/libsupc++/eh_call.cc:54:0: __cxa_call_terminate
       /data/mci/05f0e86cbdc92f18b9a81c80a01536dd/toolchain-builder/build-gcc-v2.sh-Zsd/ppc64le-mongodb-linux/libstdc++-v3/libsupc++/../../../../gcc-5.4.0/libstdc++-v3/libsupc++/eh_personality.cc:676:0: __gxx_personality_v0
       ??:0:0: ??
       ??:0:0: ??
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/third_party/boost-1.60.0/boost/optional/optional.hpp:745:0: boost::optional_detail::optional_base<mongo::AsyncRequestsSender::Response>::destroy_impl(mpl_::bool_<false>)
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/third_party/boost-1.60.0/boost/optional/optional.hpp:707:0: boost::optional_detail::optional_base<mongo::AsyncRequestsSender::Response>::destroy()
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/third_party/boost-1.60.0/boost/optional/optional.hpp:327:0: boost::optional_detail::optional_base<mongo::AsyncRequestsSender::Response>::~optional_base()
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/third_party/boost-1.60.0/boost/optional/optional.hpp:877:0: boost::optional<mongo::AsyncRequestsSender::Response>::~optional()
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/async_requests_sender.cpp:97:0: mongo::AsyncRequestsSender::next()
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/async_requests_sender.cpp:88:0: mongo::AsyncRequestsSender::~AsyncRequestsSender()
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/multi_statement_transaction_requests_sender.h:40:0: mongo::MultiStatementTransactionRequestsSender::~MultiStatementTransactionRequestsSender()
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/write_ops/batch_write_exec.cpp:217:0: mongo::BatchWriteExec::executeBatch(mongo::OperationContext*, mongo::NSTargeter&, mongo::BatchedCommandRequest const&, mongo::BatchedCommandResponse*, mongo::BatchWriteExecStats*)
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/write_ops/cluster_write.cpp:114:0: mongo::ClusterWriter::write(mongo::OperationContext*, mongo::BatchedCommandRequest const&, mongo::BatchWriteExecStats*, mongo::BatchedCommandResponse*)
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection_sharded.cpp:111:0: mongo::SessionsCollectionSharded::removeRecords(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&)::{lambda(mongo::BSONObj)#1}::operator()(mongo::BSONObj) const
       /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1857:0: std::_Function_handler<mongo::Status (mongo::BSONObj), mongo::SessionsCollectionSharded::removeRecords(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&)::{lambda(mongo::BSONObj)#1}>::_M_invoke(std::_Any_data const&, mongo::BSONObj&&)
       /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<mongo::Status (mongo::BSONObj)>::operator()(mongo::BSONObj) const
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:153:0: mongo::Status mongo::(anonymous namespace)::runBulkCmd<mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionId const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > >(mongo::StringData, mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionId const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&)::{lambda(mongo::BSONArrayBuilder)#2}::operator()(mongo::BSONArrayBuilder) const [clone .isra.170]
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:107:0: operator()
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:125:0: runBulkGeneric<mongo::(anonymous namespace)::runBulkCmd(mongo::StringData, InitBatchFn&&, AddLineFn&&, SendBatchFn&&, const Container&) [with InitBatchFn = mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONObjBuilder*)>&; AddLineFn = mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONArrayBuilder*, const mongo::LogicalSessionId&)>&; SendBatchFn = std::function<mongo::Status(mongo::BSONObj)>&; Container = std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash>]::<lambda()>, mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONArrayBuilder*, const mongo::LogicalSessionId&)>, mongo::(anonymous namespace)::runBulkCmd(mongo::StringData, InitBatchFn&&, AddLineFn&&, SendBatchFn&&, const Container&) [with InitBatchFn = mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONObjBuilder*)>&; AddLineFn = mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONArrayBuilder*, const mongo::LogicalSessionId&)>&; SendBatchFn = std::function<mongo::Status(mongo::BSONObj)>&; Container = std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash>]::<lambda(mongo::BSONArrayBuilder*)>, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash> >
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:156:0: runBulkCmd<mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONObjBuilder*)>&, mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONArrayBuilder*, const mongo::LogicalSessionId&)>&, std::function<mongo::Status(mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash> >
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:242:0: mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection_sharded.cpp:115:0: mongo::SessionsCollectionSharded::removeRecords(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&)
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/logical_session_cache_impl.cpp:351:0: mongo::LogicalSessionCacheImpl::_refresh(mongo::Client*)
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/logical_session_cache_impl.cpp:176:0: mongo::LogicalSessionCacheImpl::_periodicRefresh(mongo::Client*)
       /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void (mongo::Client*)>::operator()(mongo::Client*) const
       /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/util/periodic_runner_impl.cpp:118:0: operator()
       /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1531:0: _M_invoke<>
       /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1520:0: operator()
       /opt/mongodbtoolchain/v2/include/c++/5.4.0/thread:115:0: std::thread::_Impl<std::_Bind_simple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1} ()> >::_M_run()
       /data/mci/05f0e86cbdc92f18b9a81c80a01536dd/toolchain-builder/build-gcc-v2.sh-Zsd/ppc64le-mongodb-linux/libstdc++-v3/src/c++11/../../../../../gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84:0: execute_native_thread_routine
       ??:0:0: ??
       ??:0:0: ??
      

            Assignee:
            blake.oler@mongodb.com Blake Oler
            Reporter:
            blake.oler@mongodb.com Blake Oler
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: