[SERVER-73805] mongos in 6.0.4 hangs under load Created: 08/Feb/23 Updated: 15/Feb/23 Resolved: 15/Feb/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 6.0.4 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Minor - P4 |
| Reporter: | Amit Gupta | Assignee: | Yuan Fang |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | start mongos, wait for a while during very busy times..
|
||||||||
| Participants: | |||||||||
| Description |
|
Environment:
We just upgraded from 5.0.14 to 6.0.4. The upgrade went smoothly last night. Today during peak hours, our app servers that were getting most of our traffic would have mongos randomly hang. The mongos process looks like it is still running, however mongosh cannot connect to it.
I managed to attached gdb to the running process, and here is the backtrace:
|
| Comments |
| Comment by Yuan Fang [ 15/Feb/23 ] | |||||
|
amit.gupta@opensense.com, Glad to hear it's been resolved. I'll close this ticket now as requested. | |||||
| Comment by Amit Gupta [ 15/Feb/23 ] | |||||
|
Hi, we resynced every node in the cluster including the config servers last friday night. The cluster seems to be running smoothly since. I did not notice any dmesg or any other log entries on the server to indicate a disk problem. So I'm not sure how the data file got corrupted.
The cluster is operating smoothly right now. Feel free to close out the ticket! Thanks for looking into this. | |||||
| Comment by Yuan Fang [ 14/Feb/23 ] | |||||
|
Thank you for providing the diagnostic dataset. After reviewing it with the Triage team, the error messages lead us to suspect some form of physical corruption. The message of the Invariant failure at the first crash in the mongod.log:
The failure message after restarting from the crash also indicates the presence of possible data corruption. :
The ideal resolution is to perform a clean resync from an unaffected node. You can also try mongod --repair using the latest patch release of your version (v6.0.4) of MongoDB. In the event that a --repair operation is unsuccessful, then please also provide:
Regards, | |||||
| Comment by Amit Gupta [ 11/Feb/23 ] | |||||
|
It looks like the issue is happening in the code here: https://github.com/wiredtiger/wiredtiger/blob/develop/src/os_posix/os_fs.c
| |||||
| Comment by Amit Gupta [ 10/Feb/23 ] | |||||
|
I tried restarting the service and it looks like it crashes the same way. I uploaded the log file after restarting the crash. | |||||
| Comment by Amit Gupta [ 10/Feb/23 ] | |||||
|
Here is the log snippet in case it's helpful for other people searching for simple issue (there is more but it's too long to put in this thread. I'm uploading it separately): {"t":\{"$date":"2023-02-10T13:29:52.829-05:00"} ,"s":"E", "c":"WT", "id":22435, "ctx":"conn5691","msg":"WiredTiger error message","attr":{"error":-31802,"message": {"ts_sec":1676053792,"ts_usec":829607,"thread":"3356:0x7f534576d700","session_dhandle_name":"file:collection-97--9154580971862015175.wt","session_name":"WT_CURSOR.search","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__posix_file_read:428:/var/lib/mongo/collection-97--9154580971862015175.wt: handle-read: pread: failed to read 24576 bytes at offset 101808472064","error_str":"WT_ERROR: non-specific WiredTiger error","error_code":-31802}}} {"t": {"$date":"2023-02-10T13:29:52.829-05:00"},"s":"F", "c":"ASSERT", "id":23083, "ctx":"conn5691","msg":"Invariant failure","attr":{"expr":"seekRet","error":"UnknownError: -31802: WT_ERROR: non-specific WiredTiger error","file":"src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp","line":2281}} {"t": {"$date":"2023-02-10T13:29:52.829-05:00"},"s":"F", "c":"ASSERT", "id":23084, "ctx":"conn5691","msg":"\n\n***aborting after invariant() failure\n\n"} {"t": {"$date":"2023-02-10T13:29:52.829-05:00"},"s":"F", "c":"CONTROL", "id":6384300, "ctx":"conn5691","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31380, "ctx":"conn5691","msg":"BACKTRACE","attr":{"bt":{"backtrace":[ {"a":"55DE5BC4A364","b":"55DE56E6D000","o":"4DDD364","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"},{"a":"55DE5BC4C8A9","b":"55DE56E6D000","o":"4DDF8A9","s":"ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"},{"a":"55DE5BC46586","b":"55DE56E6D000","o":"4DD9586","s":"abruptQuit","s+":"66"},{"a":"7F5367C90420","b":"7F5367C7C000","o":"14420","s":"funlockfile","s+":"60"},{"a":"7F5367ACD00B","b":"7F5367A8A000","o":"4300B","s":"gsignal","s+":"CB"},{"a":"7F5367AAC859","b":"7F5367A8A000","o":"22859","s":"abort","s+":"12B"},{"a":"55DE58D6C851","b":"55DE56E6D000","o":"1EFF851","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j","C":"mongo::invariantOKFailed(char const*, mongo::Status const&, char const*, unsigned int)","s+":"145"},{"a":"55DE5980BB51","b":"55DE56E6D000","o":"299EB51","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase9seekExactERKNS_8RecordIdE","C":"mongo::WiredTigerRecordStoreCursorBase::seekExact(mongo::RecordId const&)","s+":"2D1"},{"a":"55DE5A12A6E8","b":"55DE56E6D000","o":"32BD6E8","s":"_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmPNS_20SeekableRecordCursorERKNS_13CollectionPtrERKNS_15NamespaceStringE","C":"mongo::WorkingSetCommon::fetch(mongo::OperationContext*, mongo::WorkingSet*, unsigned long, mongo::SeekableRecordCursor*, mongo::CollectionPtr const&, mongo::NamespaceString const&)","s+":"78"},{"a":"55DE5A0DDE84","b":"55DE56E6D000","o":"3270E84","s":"_ZN5mongo10FetchStage6doWorkEPm","C":"mongo::FetchStage::doWork(unsigned long*)","s+":"174"},{"a":"55DE5A110ABA","b":"55DE56E6D000","o":"32A3ABA","s":"_ZN5mongo16ShardFilterStage6doWorkEPm","C":"mongo::ShardFilterStage::doWork(unsigned long*)","s+":"9A"},{"a":"55DE5A0EB605","b":"55DE56E6D000","o":"327E605","s":"_ZN5mongo10LimitStage6doWorkEPm","C":"mongo::LimitStage::doWork(unsigned long*)","s+":"85"},{"a":"55DE5A0CD52D","b":"55DE56E6D000","o":"326052D","s":"_ZN5mongo9PlanStage4workEPm","C":"mongo::PlanStage::work(unsigned long*)","s+":"3D"},{"a":"55DE5A0CC44D","b":"55DE56E6D000","o":"325F44D","s":"_ZN5mongo15CachedPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE","C":"mongo::CachedPlanStage::pickBestPlan(mongo::PlanYieldPolicy*)","s+":"14D"},{"a":"55DE5A17084A","b":"55DE56E6D000","o":"330384A","s":"_ZN5mongo16PlanExecutorImpl13_pickBestPlanEv","C":"mongo::PlanExecutorImpl::_pickBestPlan()","s+":"BA"},{"a":"55DE5A171C65","b":"55DE56E6D000","o":"3304C65","s":"_ZN5mongo16PlanExecutorImplC1EPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EERKN5boost13intrusive_ptrINS_17ExpressionContextEEERKNS_13CollectionPtrEbNS_15NamespaceStringENS_15PlanYieldPolicy11YieldPolicyE","C":"mongo::PlanExecutorImpl::PlanExecutorImpl(mongo::OperationContext*, std::unique_ptr<mongo::WorkingSet, std::default_delete<mongo::WorkingSet> >, std::unique_ptr<mongo::PlanStage, std::default_delete<mongo::PlanStage> >, std::unique_ptr<mongo::QuerySolution, std::default_delete<mongo::QuerySolution> >, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::CollectionPtr const&, bool, mongo::NamespaceString, mongo::PlanYieldPolicy::YieldPolicy)","s+":"445"},{"a":"55DE5A16C9C6","b":"55DE56E6D000","o":"32FF9C6","s":"_ZN5mongo21plan_executor_factory4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EERKN5boost13intrusive_ptrINS_17ExpressionContextEEEPKNS_13CollectionPtrEmNS_15NamespaceStringENS_15PlanYieldPolicy11YieldPolicyE","C":"mongo::plan_executor_factory::make(mongo::OperationContext*, std::unique_ptr<mongo::WorkingSet, std::default_delete<mongo::WorkingSet> >, std::unique_ptr<mongo::PlanStage, std::default_delete<mongo::PlanStage> >, std::unique_ptr<mongo::QuerySolution, std::default_delete<mongo::QuerySolution> >, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::CollectionPtr const*, unsigned long, mongo::NamespaceString, mongo::PlanYieldPolicy::YieldPolicy)","s+":"106"},{"a":"55DE5A16CE83","b":"55DE56E6D000","o":"32FFE83","s":"_ZN5mongo21plan_executor_factory4makeESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS2_EES1_INS_10WorkingSetES3_IS6_EES1_INS_9PlanStageES3_IS9_EEPKNS_13CollectionPtrENS_15PlanYieldPolicy11YieldPolicyEmNS_15NamespaceStringES1_INS_13QuerySolutionES3_ISI_EE","C":"mongo::plan_executor_factory::make(std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::unique_ptr<mongo::WorkingSet, std::default_delete<mongo::WorkingSet> >, std::unique_ptr<mongo::PlanStage, std::default_delete<mongo::PlanStage> >, mongo::CollectionPtr const*, mongo::PlanYieldPolicy::YieldPolicy, unsigned long, mongo::NamespaceString, std::unique_ptr<mongo::QuerySolution, std::default_delete<mongo::QuerySolution> >)","s+":"113"},{"a":"55DE5A15AE5C","b":"55DE56E6D000","o":"32EDE5C","s":"_ZN5mongo12_GLOBAL_N_118getClassicExecutorEPNS_16OperationContextERKNS_13CollectionPtrESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS7_EENS_15PlanYieldPolicy11YieldPolicyERKNS_18QueryPlannerParamsE","C":"mongo::(anonymous namespace)::getClassicExecutor(mongo::OperationContext*, mongo::CollectionPtr const&, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, mongo::PlanYieldPolicy::YieldPolicy, mongo::QueryPlannerParams const&)","s+":"20C"},{"a":"55DE5A15B0F8","b":"55DE56E6D000","o":"32EE0F8","s":"ZN5mongo11getExecutorEPNS_16OperationContextERKNS_26MultipleCollectionAccessorESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS6_EESt8functionIFvPS6_EENS_15PlanYieldPolicy11YieldPolicyERKNS_18QueryPlannerParamsE","C":"mongo::getExecutor(mongo::OperationContext*, mongo::MultipleCollectionAccessor const&, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::function<void (mongo::CanonicalQuery*)>, mongo::PlanYieldPolicy::YieldPolicy, mongo::QueryPlannerParams const&)","s+":"148"},{"a":"55DE5A15B1FA","b":"55DE56E6D000","o":"32EE1FA","s":"_ZN5mongo15getExecutorFindEPNS_16OperationContextERKNS_26MultipleCollectionAccessorESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS6_EESt8functionIFvPS6_EEbNS_18QueryPlannerParamsE","C":"mongo::getExecutorFind(mongo::OperationContext*, mongo::MultipleCollectionAccessor const&, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::function<void (mongo::CanonicalQuery*)>, bool, mongo::QueryPlannerParams)","s+":"AA"},{"a":"55DE5A1603B1","b":"55DE56E6D000","o":"32F33B1","s":"_ZN5mongo15getExecutorFindEPNS_16OperationContextEPKNS_13CollectionPtrESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS6_EESt8functionIFvPS6_EEbm","C":"mongo::getExecutorFind(mongo::OperationContext*, mongo::CollectionPtr const*, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, std::function<void (mongo::CanonicalQuery*)>, bool, unsigned long)","s+":"F1"},{"a":"55DE59B8781C","b":"55DE56E6D000","o":"2D1A81C","s":"_ZN5mongo12_GLOBAL_N_17FindCmd10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::(anonymous namespace)::FindCmd::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)","s+":"BFC"},{"a":"55DE5A9D6591","b":"55DE56E6D000","o":"3B69591","s":"ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)","s+":"61"},{"a":"55DE5A9DA609","b":"55DE56E6D000","o":"3B6D609","s":"_ZN5mongo14CommandHelpers20runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEENS_9transport15ServiceExecutor14ThreadingModelE","C":"mongo::CommandHelpers::runCommandInvocation(std::shared_ptr<mongo::RequestExecutionContext>, std::shared_ptr<mongo::CommandInvocation>, mongo::transport::ServiceExecutor::ThreadingModel)","s+":"1C9"},{"a":"55DE58FD4A47","b":"55DE56E6D000","o":"2167A47","s":"_ZN5mongo12_GLOBAL_N_120runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEE","C":"mongo::(anonymous namespace)::runCommandInvocation(std::shared_ptr<mongo::RequestExecutionContext>, std::shared_ptr<mongo::CommandInvocation>)","s+":"97"},{"a":"55DE58FE4B46","b":"55DE56E6D000","o":"2177B46","s":"ZN5mongo12_GLOBAL_N_114RunCommandImpl11_runCommandEv","C":"mongo::(anonymous namespace)::RunCommandImpl::runCommand()","s+":"146"},{"a":"55DE58FE5B1B","b":"55DE56E6D000","o":"2178B1B","s":"_ZN5mongo12_GLOBAL_N_114RunCommandImpl8_runImplEv","C":"mongo::(anonymous namespace)::RunCommandImpl::runImpl()","s+":"8B"},{"a":"55DE58FD5AFC","b":"55DE56E6D000","o":"2168AFC","s":"_ZN5mongo12_GLOBAL_N_114RunCommandImpl3runEv","C":"mongo::(anonymous namespace)::RunCommandImpl::run()","s+":"13C"},{"a":"55DE58FDCC3E","b":"55DE56E6D000","o":"216FC3E","s":"ZN5mongo12_GLOBAL_N_119ExecCommandDatabase12_commandExecEv","C":"mongo::(anonymous namespace)::ExecCommandDatabase::commandExec()","s+":"34E"},{"a":"55DE58FE177F","b":"55DE56E6D000","o":"217477F","s":"_ZZN5mongo12_GLOBAL_N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEEENUlvE0_clEv","C":"mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>):: {lambda()#2}::operator()()","s+":"43F"},{"a":"55DE58FE3370","b":"55DE56E6D000","o":"2176370","s":"ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL_N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE0_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","C":"mongo::unique_function<void (mongo::future_details::SharedStateBase*)>::makeImpl<mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::makeContinuation<void, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::makeContinuation<void, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1})::SpecificImpl::call(mongo::future_details::SharedStateBase*&&)","s+":"80"},{"a":"55DE58F93EA7","b":"55DE56E6D000","o":"2126EA7","s":"ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","C":"mongo::future_details::SharedStateBase::transitionToFinished()","s+":"147"},{"a":"55DE58FE72D2","b":"55DE56E6D000","o":"217A2D2","s":"_ZN5mongo14future_details10FutureImplINS0_8FakeVoidEE11generalImplIZNOS3_17propagateResultToEPNS0_15SharedStateImplIS2_EEEUlOS2_E_ZNOS3_17propagateResultToES7_EUlONS_6StatusEE0_ZNOS3_17propagateResultToES7_EUlvE1_EEDaOT_OT0_OT1","C":"auto mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::generalImpl<mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>*) &&:: {lambda(mongo::future_details::FakeVoid&&)#1}, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda(mongo::Status&&)#2}, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda()#3}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda(mongo::future_details::FakeVoid&&)#1}&&, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda(mongo::Status&&)#2}&&, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>) &&::{lambda()#3}&&)","s+":"152"},{"a":"55DE58FD7C3C","b":"55DE56E6D000","o":"216AC3C","s":"ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL_N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","C":"mongo::unique_function<void (mongo::future_details::SharedStateBase)>::makeImpl<mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::makeContinuation<void, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>):: {lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::makeContinuation<void, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::then<mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1})::SpecificImpl::call(mongo::future_details::SharedStateBase*&&)","s+":"AC"},{"a":"55DE58F93EA7","b":"55DE56E6D000","o":"2126EA7","s":"ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","C":"mongo::future_details::SharedStateBase::transitionToFinished()","s+":"147"},{"a":"55DE58FE2103","b":"55DE56E6D000","o":"2175103","s":"_ZN5mongo12_GLOBAL_N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","C":"mongo::(anonymous namespace)::executeCommand(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)","s+":"1D3"},{"a":"55DE58FE2975","b":"55DE56E6D000","o":"2175975","s":"ZN5mongo12_GLOBAL_N_116receivedCommandsESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","C":"mongo::(anonymous namespace)::receivedCommands(std::shared_ptr<mongo::(anonymous namespace)::HandleRequest::ExecutionContext>)","s+":"2D5"},{"a":"55DE58FE32B6","b":"55DE56E6D000","o":"21762B6","s":"ZN5mongo12_GLOBAL_N_115CommandOpRunner3runEv","C":"mongo::(anonymous namespace)::CommandOpRunner::run()","s+":"36"},{"a":"55DE58FDA28F","b":"55DE56E6D000","o":"216D28F","s":"ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageESt10unique_ptrIKNS0_5HooksESt14default_deleteIS8_EE","C":"mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, std::unique_ptr<mongo::ServiceEntryPointCommon::Hooks const, std::default_delete<mongo::ServiceEntryPointCommon::Hooks const> >)","s+":"53F"},{"a":"55DE58FCEA4F","b":"55DE56E6D000","o":"2161A4F","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","C":"mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)","s+":"4F"},{"a":"55DE59E34027","b":"55DE56E6D000","o":"2FC7027","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14processMessageEv","C":"mongo::transport::ServiceStateMachine::Impl::processMessage()","s+":"127"},{"a":"55DE59E35D36","b":"55DE56E6D000","o":"2FC8D36","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","C":"mongo::transport::ServiceStateMachine::Impl::startNewLoop(mongo::Status const&)","s+":"76"},{"a":"55DE59E361CF","b":"55DE56E6D000","o":"2FC91CF","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1","C":"mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::transport::ServiceStateMachine::Impl::scheduleNewLoop(mongo::Status)::{lambda(mongo::Status)#1}>(mongo::transport::ServiceStateMachine::Impl::scheduleNewLoop(mongo::Status)::{lambda(mongo::Status)#1}&&)::SpecificImpl::call(mongo::Status&&)","s+":"7F"},{"a":"55DE5B44B2BD","b":"55DE56E6D000","o":"45DE2BD","s":"ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport26ServiceExecutorSynchronous18runOnDataAvailableERKSt10shared_ptrINS5_7SessionEES3_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1","C":"mongo::unique_function<void (mongo::Status)>::makeImpl<mongo::transport::ServiceExecutorSynchronous::runOnDataAvailable(std::shared_ptr<mongo::transport::Session> const&, mongo::unique_function<void (mongo::Status)>)::{lambda(mongo::Status)#1}>(mongo::transport::ServiceExecutorSynchronous::runOnDataAvailable(std::shared_ptr<mongo::transport::Session> const&, mongo::unique_function<void (mongo::Status)>)::{lambda(mongo::Status)#1}&&)::SpecificImpl::call(mongo::Status&&)","s+":"3D"},{"a":"55DE595217D3","b":"55DE56E6D000","o":"26B47D3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function<void ()>::makeImpl<mongo::transport::ServiceExecutor::schedule(mongo::unique_function<void (mongo::Status)>)::{lambda()#1}>(mongo::transport::ServiceExecutor::schedule(mongo::unique_function<void (mongo::Status)>)::{lambda()#1}&&)::SpecificImpl::call()","s+":"33"},{"a":"55DE5B44C34B","b":"55DE56E6D000","o":"45DF34B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function<void ()>::makeImpl<mongo::transport::ServiceExecutorSynchronous::scheduleTask(mongo::unique_function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags):: {lambda()#2}>(mongo::transport::ServiceExecutorSynchronous::scheduleTask(mongo::unique_function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags)::{lambda()#2}&&)::SpecificImpl::call()","s+":"BB"},{"a":"55DE5B44E16C","b":"55DE56E6D000","o":"45E116C","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function<void ()>::makeImpl<mongo::launchServiceWorkerThread(mongo::unique_function<void ()>):: {lambda()#4}>(mongo::launchServiceWorkerThread(mongo::unique_function<void ()>)::{lambda()#4}&&)::SpecificImpl::call()","s+":"5C"},{"a":"55DE5B44E1DC","b":"55DE56E6D000","o":"45E11DC","s":"ZN5mongo12_GLOBAL_N_17runFuncEPv","C":"mongo::(anonymous namespace)::runFunc(void*)","s+":"1C"},{"a":"7F5367C84609","b":"7F5367C7C000","o":"8609","s":"start_thread","s+":"D9"},{"a":"7F5367BA9133","b":"7F5367A8A000","o":"11F133","s":"clone","s+":"43"}],"processInfo":{"mongodbVersion":"6.0.4","gitVersion":"44ff59461c1353638a71e710f385a566bcd2f547","compiledModules":[],"uname": {"sysname":"Linux","release":"5.4.0-139-generic","version":"#156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023","machine":"x86_64"},"somap":[\{"b":"55DE56E6D000","elfType":3,"buildId":"C934FA633D74614303D55DB3674B3E4C19C8AAD6"},\{"b":"7F5367C7C000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"7B4536F41CDAA5888408E82D0836E33DCF436466"},\{"b":"7F5367A8A000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"1878E6B475720C7C51969E69AB2D276FAE6D1DEE"}]}}},"tags":[]} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5BC4A364","b":"55DE56E6D000","o":"4DDD364","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5BC4C8A9","b":"55DE56E6D000","o":"4DDF8A9","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5BC46586","b":"55DE56E6D000","o":"4DD9586","s":"abruptQuit","s+":"66"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"7F5367C90420","b":"7F5367C7C000","o":"14420","s":"funlockfile","s+":"60"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"7F5367ACD00B","b":"7F5367A8A000","o":"4300B","s":"gsignal","s+":"CB"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"7F5367AAC859","b":"7F5367A8A000","o":"22859","s":"abort","s+":"12B"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE58D6C851","b":"55DE56E6D000","o":"1EFF851","s":"_ZN5mongo17invariantOKFailedEPKcRKNS_6StatusES1_j","C":"mongo::invariantOKFailed(char const*, mongo::Status const&, char const*, unsigned int)","s+":"145"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5980BB51","b":"55DE56E6D000","o":"299EB51","s":"_ZN5mongo31WiredTigerRecordStoreCursorBase9seekExactERKNS_8RecordIdE","C":"mongo::WiredTigerRecordStoreCursorBase::seekExact(mongo::RecordId const&)","s+":"2D1"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5A12A6E8","b":"55DE56E6D000","o":"32BD6E8","s":"_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmPNS_20SeekableRecordCursorERKNS_13CollectionPtrERKNS_15NamespaceStringE","C":"mongo::WorkingSetCommon::fetch(mongo::OperationContext*, mongo::WorkingSet*, unsigned long, mongo::SeekableRecordCursor*, mongo::CollectionPtr const&, mongo::NamespaceString const&)","s+":"78"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5A0DDE84","b":"55DE56E6D000","o":"3270E84","s":"_ZN5mongo10FetchStage6doWorkEPm","C":"mongo::FetchStage::doWork(unsigned long*)","s+":"174"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5A110ABA","b":"55DE56E6D000","o":"32A3ABA","s":"_ZN5mongo16ShardFilterStage6doWorkEPm","C":"mongo::ShardFilterStage::doWork(unsigned long*)","s+":"9A"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5A0EB605","b":"55DE56E6D000","o":"327E605","s":"_ZN5mongo10LimitStage6doWorkEPm","C":"mongo::LimitStage::doWork(unsigned long*)","s+":"85"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5A0CD52D","b":"55DE56E6D000","o":"326052D","s":"_ZN5mongo9PlanStage4workEPm","C":"mongo::PlanStage::work(unsigned long*)","s+":"3D"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5A0CC44D","b":"55DE56E6D000","o":"325F44D","s":"_ZN5mongo15CachedPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE","C":"mongo::CachedPlanStage::pickBestPlan(mongo::PlanYieldPolicy*)","s+":"14D"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5A17084A","b":"55DE56E6D000","o":"330384A","s":"_ZN5mongo16PlanExecutorImpl13_pickBestPlanEv","C":"mongo::PlanExecutorImpl::_pickBestPlan()","s+":"BA"}}} {"t": {"$date":"2023-02-10T13:29:53.035-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn5691","msg":"Frame","attr":{"frame": {"a":"55DE5A171C65","b":"55DE56E6D000","o":"3304C65","s":"_ZN5mongo16PlanExecutorImplC1EPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EERKN5boost13intrusive_ptrINS_17ExpressionContextEEERKNS_13CollectionPtrEbNS_15NamespaceStringENS_15PlanYieldPolicy11YieldPolicyE","C":"mongo::PlanExecutorImpl::PlanExecutorImpl(mongo::OperationContext*, std::unique_ptr<mongo::WorkingSet, std::default_delete<mongo::WorkingSet> >, std::unique_ptr<mongo::PlanStage, std::default_delete<mongo::PlanStage> >, std::unique_ptr<mongo::QuerySolution, std::default_delete<mongo::QuerySolution> >, std::unique_ptr<mongo::CanonicalQuery, std::default_delete<mongo::CanonicalQuery> >, boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::CollectionPtr const&, bool, mongo::NamespaceString, mongo::PlanYieldPolicy::YieldPolicy)","s+":"445"}}} | |||||
| Comment by Amit Gupta [ 10/Feb/23 ] | |||||
|
So we had a different crash today on one node in the 3rd shared of our cluster.
I'm attaching the diagnostic data as well as the full mongod log. Here is the output of systemctl status mongod:
● mongod.service - MongoDB Database Server Feb 08 18:18:34 pndb3.sendergen.com systemd[1]: Started MongoDB Database Server. ,"s":"I", "c":"CONTROL", "id":5760901, "ctx":"-","msg":"Applied --setParameter options","attr":{"serverParameters":{"cursorTimeoutMillis":{"default":600000,"value":60000000}}}}
| |||||
| Comment by Yuan Fang [ 09/Feb/23 ] | |||||
|
Thank you for getting us the diagnostic data! In the meantime, we looked at the backtrace you provided in this ticket (and Regards, | |||||
| Comment by Amit Gupta [ 09/Feb/23 ] | |||||
|
Hi Yuan, this issue is probably unrelated to
Will get you the logs requested soon. | |||||
| Comment by Yuan Fang [ 09/Feb/23 ] | |||||
|
Thank you for your report. We need additional information to diagnose the problem. I've created a secure upload portal for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time. For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link:
Regards, |