[SERVER-37404] Terminate called on AsyncRequestsSender destructor Created: 01/Oct/18  Updated: 08/Jan/24  Resolved: 03/Oct/18

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: None
Fix Version/s: 4.1.4

Type: Bug Priority: Major - P3
Reporter: Blake Oler Assignee: Blake Oler
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-37430 Destroy sharding task executors and A... Closed
is depended on by SERVER-37330 Add sharded passthrough suites to det... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Sharding 2018-10-08
Participants:

 Description   

In logical session cache sharding refresh suites, we occasionally see terminate called from the AsyncRequestsSender on shutdown.

[ShardedClusterFixture:job3:configsvr:primary] 2018-09-30T20:43:46.127+0000 F -        [LogicalSessionCacheRefresh] terminate() called. An exception is active; attempting to gather more information
[ShardedClusterFixture:job3:configsvr:primary] 2018-09-30T20:43:46.142+0000 F -        [LogicalSessionCacheRefresh] DBException::toString(): InterruptedAtShutdown: interrupted at shutdown
[ShardedClusterFixture:job3:configsvr:primary] Actual exception type: mongo::error_details::ExceptionForImpl<(mongo::ErrorCodes::Error)11600, mongo::ExceptionForCat<(mongo::ErrorCategory)1>, mongo::ExceptionForCat<(mongo::ErrorCategory)6> >

Lobster

Symbolized backtrace:

/data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/util/stacktrace_posix.cpp:172:0: mongo::printStackTrace(std::ostream&)
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/util/signal_handlers_synchronous.cpp:229:0: mongo::(anonymous namespace)::myTerminate()
 /data/mci/05f0e86cbdc92f18b9a81c80a01536dd/toolchain-builder/build-gcc-v2.sh-Zsd/ppc64le-mongodb-linux/libstdc++-v3/libsupc++/../../../../gcc-5.4.0/libstdc++-v3/libsupc++/eh_terminate.cc:47:0: __cxxabiv1::__terminate(void (*)())
 /data/mci/05f0e86cbdc92f18b9a81c80a01536dd/toolchain-builder/build-gcc-v2.sh-Zsd/ppc64le-mongodb-linux/libstdc++-v3/libsupc++/../../../../gcc-5.4.0/libstdc++-v3/libsupc++/eh_call.cc:54:0: __cxa_call_terminate
 /data/mci/05f0e86cbdc92f18b9a81c80a01536dd/toolchain-builder/build-gcc-v2.sh-Zsd/ppc64le-mongodb-linux/libstdc++-v3/libsupc++/../../../../gcc-5.4.0/libstdc++-v3/libsupc++/eh_personality.cc:676:0: __gxx_personality_v0
 ??:0:0: ??
 ??:0:0: ??
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/third_party/boost-1.60.0/boost/optional/optional.hpp:745:0: boost::optional_detail::optional_base<mongo::AsyncRequestsSender::Response>::destroy_impl(mpl_::bool_<false>)
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/third_party/boost-1.60.0/boost/optional/optional.hpp:707:0: boost::optional_detail::optional_base<mongo::AsyncRequestsSender::Response>::destroy()
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/third_party/boost-1.60.0/boost/optional/optional.hpp:327:0: boost::optional_detail::optional_base<mongo::AsyncRequestsSender::Response>::~optional_base()
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/third_party/boost-1.60.0/boost/optional/optional.hpp:877:0: boost::optional<mongo::AsyncRequestsSender::Response>::~optional()
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/async_requests_sender.cpp:97:0: mongo::AsyncRequestsSender::next()
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/async_requests_sender.cpp:88:0: mongo::AsyncRequestsSender::~AsyncRequestsSender()
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/multi_statement_transaction_requests_sender.h:40:0: mongo::MultiStatementTransactionRequestsSender::~MultiStatementTransactionRequestsSender()
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/write_ops/batch_write_exec.cpp:217:0: mongo::BatchWriteExec::executeBatch(mongo::OperationContext*, mongo::NSTargeter&, mongo::BatchedCommandRequest const&, mongo::BatchedCommandResponse*, mongo::BatchWriteExecStats*)
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/s/write_ops/cluster_write.cpp:114:0: mongo::ClusterWriter::write(mongo::OperationContext*, mongo::BatchedCommandRequest const&, mongo::BatchWriteExecStats*, mongo::BatchedCommandResponse*)
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection_sharded.cpp:111:0: mongo::SessionsCollectionSharded::removeRecords(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&)::{lambda(mongo::BSONObj)#1}::operator()(mongo::BSONObj) const
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1857:0: std::_Function_handler<mongo::Status (mongo::BSONObj), mongo::SessionsCollectionSharded::removeRecords(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&)::{lambda(mongo::BSONObj)#1}>::_M_invoke(std::_Any_data const&, mongo::BSONObj&&)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<mongo::Status (mongo::BSONObj)>::operator()(mongo::BSONObj) const
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:153:0: mongo::Status mongo::(anonymous namespace)::runBulkCmd<mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionId const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > >(mongo::StringData, mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONObjBuilder*)#1}&, mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)::{lambda(mongo::BSONArrayBuilder*, mongo::LogicalSessionId const&)#2}&, std::function<mongo::Status (mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&)::{lambda(mongo::BSONArrayBuilder)#2}::operator()(mongo::BSONArrayBuilder) const [clone .isra.170]
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:107:0: operator()
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:125:0: runBulkGeneric<mongo::(anonymous namespace)::runBulkCmd(mongo::StringData, InitBatchFn&&, AddLineFn&&, SendBatchFn&&, const Container&) [with InitBatchFn = mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONObjBuilder*)>&; AddLineFn = mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONArrayBuilder*, const mongo::LogicalSessionId&)>&; SendBatchFn = std::function<mongo::Status(mongo::BSONObj)>&; Container = std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash>]::<lambda()>, mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONArrayBuilder*, const mongo::LogicalSessionId&)>, mongo::(anonymous namespace)::runBulkCmd(mongo::StringData, InitBatchFn&&, AddLineFn&&, SendBatchFn&&, const Container&) [with InitBatchFn = mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONObjBuilder*)>&; AddLineFn = mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONArrayBuilder*, const mongo::LogicalSessionId&)>&; SendBatchFn = std::function<mongo::Status(mongo::BSONObj)>&; Container = std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash>]::<lambda(mongo::BSONArrayBuilder*)>, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash> >
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:156:0: runBulkCmd<mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONObjBuilder*)>&, mongo::SessionsCollection::doRemove(const mongo::NamespaceString&, const LogicalSessionIdSet&, mongo::SessionsCollection::SendBatchFn)::<lambda(mongo::BSONArrayBuilder*, const mongo::LogicalSessionId&)>&, std::function<mongo::Status(mongo::BSONObj)>&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash> >
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection.cpp:242:0: mongo::SessionsCollection::doRemove(mongo::NamespaceString const&, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&, std::function<mongo::Status (mongo::BSONObj)>)
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/sessions_collection_sharded.cpp:115:0: mongo::SessionsCollectionSharded::removeRecords(mongo::OperationContext*, std::unordered_set<mongo::LogicalSessionId, mongo::LogicalSessionIdHash, std::equal_to<mongo::LogicalSessionId>, std::allocator<mongo::LogicalSessionId> > const&)
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/logical_session_cache_impl.cpp:351:0: mongo::LogicalSessionCacheImpl::_refresh(mongo::Client*)
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/db/logical_session_cache_impl.cpp:176:0: mongo::LogicalSessionCacheImpl::_periodicRefresh(mongo::Client*)
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:2267:0: std::function<void (mongo::Client*)>::operator()(mongo::Client*) const
 /data/mci/b4f93f1e4ad882f393cdb5b45a7e0150/src/src/mongo/util/periodic_runner_impl.cpp:118:0: operator()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1531:0: _M_invoke<>
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/functional:1520:0: operator()
 /opt/mongodbtoolchain/v2/include/c++/5.4.0/thread:115:0: std::thread::_Impl<std::_Bind_simple<mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#1} ()> >::_M_run()
 /data/mci/05f0e86cbdc92f18b9a81c80a01536dd/toolchain-builder/build-gcc-v2.sh-Zsd/ppc64le-mongodb-linux/libstdc++-v3/src/c++11/../../../../../gcc-5.4.0/libstdc++-v3/src/c++11/thread.cc:84:0: execute_native_thread_routine
 ??:0:0: ??
 ??:0:0: ??



 Comments   
Comment by Blake Oler [ 03/Oct/18 ]

Fixed by SERVER-37430.

Comment by Mira Carey [ 02/Oct/18 ]

It looks like this is a destruction ordering problem that's unique to mongod.

Specfically https://github.com/mongodb/mongo/blob/aaa64733625433f31a5e212f4652a58816385388/src/mongo/db/db.cpp#L901-L904

Note that the periodic runner is shutdown after the grid's executor pools is shutdown. This causes the sessions collection refresh code (which uses the ARS) to call cancel() on an in shutdown pool, which throws. And those throws occur out of a destructor (in the ARS)

We do shutdown the other way in mongos, so it doesn't show up there.

Generated at Thu Feb 08 04:45:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.