[SERVER-58113] mongoDB crash after Invariant failure in btree_key_generator.cpp Created: 28/Jun/21  Updated: 16/Dec/21  Resolved: 16/Dec/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.4.6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Viktor Smirnov Assignee: Mihai Andrei
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File mongoInvariantFailure.txt     Text File mongodb_4.4.8_ubuntu_20.04.2_invariant_failure.log    
Operating System: ALL
Sprint: Query Execution 2021-07-26, QE 2021-08-09, QE 2021-08-23, QE 2021-09-06, QE 2021-09-20, QE 2021-10-18, QE 2021-11-01, QE 2021-11-15, QE 2021-11-29, QE 2021-12-13, QE 2021-12-27
Participants:

 Description   

We have upgraded our mongoDB replica set members from version 3.2 to 4.4.6. Today two members of the replica crashed with following records in logs:

Server 1

{"t":{"$date":"2021-06-28T04:10:06.789+00:00"},"s":"F", "c":"-", "id":23079, "ctx":"conn11630","msg":"Invariant failure","attr":{"expr":"elt.type() != BSONType::Array","file":"src/mongo/db/index/btree_key_generator.cpp","line":75}}
{"t":{"$date":"2021-06-28T04:10:06.790+00:00"},"s":"F", "c":"-", "id":23080, "ctx":"conn11630","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2021-06-28T04:10:06.814+00:00"},"s":"F", "c":"CONTROL", "id":4757800, "ctx":"conn11630","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31431, "ctx":"conn11630","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"55E94C713C1A","b":"55E949987000","o":"2D8CC1A","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"},{"a":"55E94C715699","b":"55E949987000","o":"2D8E699","s":"_ZN5mongo15printStackTraceEv","s+":"29"},{"a":"55E94C712A46","b":"55E949987000","o":"2D8BA46","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"},{"a":"7F1D0827E980","b":"7F1D0826C000","o":"12980","s":"funlockfile","s+":"50"},{"a":"7F1D07EB9FB7","b":"7F1D07E7B000","o":"3EFB7","s":"gsignal","s+":"C7"},{"a":"7F1D07EBB921","b":"7F1D07E7B000","o":"40921","s":"abort","s+":"141"},{"a":"55E94A8B42E7","b":"55E949987000","o":"F2D2E7","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"12C"},{"a":"55E94A76DF6F","b":"55E949987000","o":"DE6F6F","s":"_ZN5mongo12_GLOBAL__N_128extractNonArrayElementAtPathERKNS_7BSONObjENS_10StringDataE.cold.338","s+":"2E"},{"a":"55E94B74116E","b":"55E949987000","o":"1DBA16E","s":"_ZNK5mongo17BtreeKeyGenerator20_getKeysWithoutArrayERKNS_7BSONObjEN5boost8optionalINS_8RecordIdEEEPSt3setINS_9KeyString5ValueESt4lessISA_ESaISA_EE","s+":"1CE"},{"a":"55E94B742E2F","b":"55E949987000","o":"1DBBE2F","s":"_ZNK5mongo17BtreeKeyGenerator7getKeysERKNS_7BSONObjEbPSt3setINS_9KeyString5ValueESt4lessIS6_ESaIS6_EEPSt6vectorIS4_ImS7_ImESaImEESaISF_EEN5boost8optionalINS_8RecordIdEEE","s+":"9F"},{"a":"55E94B383D4E","b":"55E949987000","o":"19FCD4E","s":"_ZNK5mongo17BtreeAccessMethod9doGetKeysERKNS_7BSONObjENS_17IndexAccessMethod14GetKeysContextEPSt3setINS_9KeyString5ValueESt4lessIS8_ESaIS8_EESD_PSt6vectorIS6_ImS9_ImESaImEESaISH_EEN5boost8optionalINS_8RecordIdEEE","s+":"4E"},{"a":"55E94B459BFA","b":"55E949987000","o":"1AD2BFA","s":"_ZNK5mongo25AbstractIndexAccessMethod7getKeysERKNS_7BSONObjENS_17IndexAccessMethod11GetKeysModeENS4_14GetKeysContextEPSt3setINS_9KeyString5ValueESt4lessIS9_ESaIS9_EESE_PSt6vectorIS7_ImSA_ImESaImEESaISI_EEN5boost8optionalINS_8RecordIdEEESt8functionIFvNS_6StatusES3_SP_EE","s+":"8A"},{"a":"55E94B3F800B","b":"55E949987000","o":"1A7100B","s":"_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmNS_11unowned_ptrINS_20SeekableRecordCursorEEERKNS_15NamespaceStringE","s+":"7AB"},{"a":"55E94B3C0166","b":"55E949987000","o":"1A39166","s":"_ZN5mongo10FetchStage6doWorkEPm","s+":"136"},{"a":"55E94B3D6F14","b":"55E949987000","o":"1A4FF14","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"},{"a":"55E94B3D5B1B","b":"55E949987000","o":"1A4EB1B","s":"_ZN5mongo7OrStage6doWorkEPm","s+":"8B"},{"a":"55E94B3D6F14","b":"55E949987000","o":"1A4FF14","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"},{"a":"55E94B3C0089","b":"55E949987000","o":"1A39089","s":"_ZN5mongo10FetchStage6doWorkEPm","s+":"59"},{"a":"55E94B3D6F14","b":"55E949987000","o":"1A4FF14","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"},{"a":"55E94B3D0196","b":"55E949987000","o":"1A49196","s":"_ZN5mongo14MultiPlanStage12workAllPlansEmPNS_15PlanYieldPolicyE","s+":"136"},{"a":"55E94B3D0857","b":"55E949987000","o":"1A49857","s":"_ZN5mongo14MultiPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE","s+":"A7"},{"a":"55E94B41E302","b":"55E949987000","o":"1A97302","s":"_ZN5mongo16PlanExecutorImpl13_pickBestPlanEv","s+":"92"},{"a":"55E94B42099A","b":"55E949987000","o":"1A9999A","s":"_ZN5mongo16PlanExecutorImpl4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EERKN5boost13intrusive_ptrINS_17ExpressionContextEEEPKNS_10CollectionENS_15NamespaceStringENS_12PlanExecutor11YieldPolicyE","s+":"18A"},{"a":"55E94B4211BC","b":"55E949987000","o":"1A9A1BC","s":"_ZN5mongo12PlanExecutor4makeESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS2_EES1_INS_10WorkingSetES3_IS6_EES1_INS_9PlanStageES3_IS9_EEPKNS_10CollectionENS0_11YieldPolicyENS_15NamespaceStringES1_INS_13QuerySolutionES3_ISH_EE","s+":"10C"},{"a":"55E94B415131","b":"55E949987000","o":"1A8E131","s":"_ZN5mongo11getExecutorEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EENS_12PlanExecutor11YieldPolicyEm","s+":"151"},{"a":"55E94B41529F","b":"55E949987000","o":"1A8E29F","s":"_ZN5mongo12_GLOBAL__N_116_getExecutorFindEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS6_EENS_12PlanExecutor11YieldPolicyEm","s+":"6F"},{"a":"55E94B415331","b":"55E949987000","o":"1A8E331","s":"_ZN5mongo15getExecutorFindEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EEbm","s+":"51"},{"a":"55E94B40086C","b":"55E949987000","o":"1A7986C","s":"_ZN5mongo12_GLOBAL__N_120attemptToGetExecutorERKN5boost13intrusive_ptrINS_17ExpressionContextEEEPNS_10CollectionERKNS_15NamespaceStringENS_7BSONObjESC_RKSt6bitsetILm11EESC_NS1_8optionalIxEENSH_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPKNS_18AggregationRequestEmRKy.isra.821","s+":"73C"},{"a":"55E94B400F3B","b":"55E949987000","o":"1A79F3B","s":"_ZN5mongo9PipelineD15prepareExecutorERKN5boost13intrusive_ptrINS_17ExpressionContextEEEPNS_10CollectionERKNS_15NamespaceStringEPNS_8PipelineERKNS2_INS_18DocumentSourceSortEEESt10unique_ptrINS_36GroupFromFirstDocumentTransformationESt14default_deleteISJ_EESt6bitsetILm11EERKNS_7BSONObjENS1_8optionalIxEEPKNS_18AggregationRequestERKyPb","s+":"55B"},{"a":"55E94B4014F5","b":"55E949987000","o":"1A7A4F5","s":"_ZN5mongo9PipelineD30buildInnerQueryExecutorGenericEPNS_10CollectionERKNS_15NamespaceStringEPKNS_18AggregationRequestEPNS_8PipelineE","s+":"235"},{"a":"55E94B402C8A","b":"55E949987000","o":"1A7BC8A","s":"_ZN5mongo9PipelineD23buildInnerQueryExecutorEPNS_10CollectionERKNS_15NamespaceStringEPKNS_18AggregationRequestEPNS_8PipelineE","s+":"54A"},{"a":"55E94B0A0FFD","b":"55E949987000","o":"1719FFD","s":"_ZN5mongo12runAggregateEPNS_16OperationContextERKNS_15NamespaceStringERKNS_18AggregationRequestERKNS_18LiteParsedPipelineERKNS_7BSONObjERKSt6vectorINS_9PrivilegeESaISF_EEPNS_3rpc21ReplyBuilderInterfaceE","s+":"284D"},{"a":"55E94B0987BC","b":"55E949987000","o":"17117BC","s":"_ZN5mongo12_GLOBAL__N_115PipelineCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","s+":"6C"},{"a":"55E94B66985F","b":"55E949987000","o":"1CE285F","s":"_ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","s+":"7F"},{"a":"55E94ADBBBEF","b":"55E949987000","o":"1434BEF","s":"_ZN5mongo12_GLOBAL__N_114runCommandImplEPNS_16OperationContextEPNS_17CommandInvocationERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceENS_11LogicalTimeERKNS_23ServiceEntryPointCommon5HooksEPNS_14BSONObjBuilderERKNS_30OperationSessionInfoFromClientE","s+":"9FF"},{"a":"55E94ADBEC3E","b":"55E949987000","o":"1437C3E","s":"_ZZN5mongo12_GLOBAL__N_116receivedCommandsEPNS_16OperationContextERKNS_7MessageERKNS_23ServiceEntryPointCommon5HooksEENKUlvE_clEv","s+":"1C9E"},{"a":"55E94ADBFE7D","b":"55E949987000","o":"1438E7D","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE","s+":"51D"},{"a":"55E94ADADACC","b":"55E949987000","o":"1426ACC","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","s+":"3C"},{"a":"55E94ADB8A32","b":"55E949987000","o":"1431A32","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE","s+":"F2"},{"a":"55E94ADB6775","b":"55E949987000","o":"142F775","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"125"},{"a":"55E94ADB7606","b":"55E949987000","o":"1430606","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"},{"a":"55E94C0D584A","b":"55E949987000","o":"274E84A","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE","s+":"18A"},{"a":"55E94ADB5A7B","b":"55E949987000","o":"142EA7B","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE","s+":"DB"},{"a":"55E94ADB6D4D","b":"55E949987000","o":"142FD4D","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE","s+":"50D"},{"a":"55E94ADB6E30","b":"55E949987000","o":"142FE30","s":"_ZN5mongo14future_details4callIRZNS_19ServiceStateMachine14_sourceMessageENS2_11ThreadGuardEEUlNS_10StatusWithINS_7MessageEEEE0_S6_EEDaOT_OT0_","s+":"60"},{"a":"55E94ADB70D5","b":"55E949987000","o":"14300D5","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE","s+":"145"},{"a":"55E94ADB67EA","b":"55E949987000","o":"142F7EA","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"19A"},{"a":"55E94ADB7606","b":"55E949987000","o":"1430606","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"},{"a":"55E94C0D5EC4","b":"55E949987000","o":"274EEC4","s":"_ZNSt17_Function_handlerIFvvEZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIS0_ENS2_15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameEEUlvE0_E9_M_invokeERKSt9_Any_data","s+":"D4"},{"a":"55E94C426F06","b":"55E949987000","o":"2A9FF06","s":"_ZNSt17_Function_handlerIFvvEZN5mongo25launchServiceWorkerThreadESt8functionIS0_EEUlvE1_E9_M_invokeERKSt9_Any_data","s+":"56"},{"a":"55E94C426F74","b":"55E949987000","o":"2A9FF74","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"14"},{"a":"7F1D082736DB","b":"7F1D0826C000","o":"76DB","s":"start_thread","s+":"DB"},{"a":"7F1D07F9C71F","b":"7F1D07E7B000","o":"12171F","s":"clone","s+":"3F"}],"processInfo":{"mongodbVersion":"4.4.6","gitVersion":"72e66213c2c3eab37d9358d5e78ad7f5c1d0d0d7","compiledModules":[],"uname":{"sysname":"Linux","release":"5.4.0-1046-gcp","version":"#49~18.04.1-Ubuntu SMP Fri Jun 18 21:33:55 UTC 2021","machine":"x86_64"},"somap":[{"b":"55E949987000","elfType":3,"buildId":"9E4C4D3B751D15533C310C2C0215E384501A5743"},{"b":"7F1D0826C000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"68F36706EB2E6EEE4046C4FDCA2A19540B2F6113"},{"b":"7F1D07E7B000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"CE450EB01A5E5ACC7CE7B8C2633B02CC1093339E"}]}}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94C713C1A","b":"55E949987000","o":"2D8CC1A","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94C715699","b":"55E949987000","o":"2D8E699","s":"_ZN5mongo15printStackTraceEv","s+":"29"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94C712A46","b":"55E949987000","o":"2D8BA46","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9siginfo_tPv","s+":"66"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"7F1D0827E980","b":"7F1D0826C000","o":"12980","s":"funlockfile","s+":"50"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"7F1D07EB9FB7","b":"7F1D07E7B000","o":"3EFB7","s":"gsignal","s+":"C7"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"7F1D07EBB921","b":"7F1D07E7B000","o":"40921","s":"abort","s+":"141"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94A8B42E7","b":"55E949987000","o":"F2D2E7","s":"_ZN5mongo15invariantFailedEPKcS1_j","s+":"12C"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94A76DF6F","b":"55E949987000","o":"DE6F6F","s":"_ZN5mongo12_GLOBAL__N_128extractNonArrayElementAtPathERKNS_7BSONObjENS_10StringDataE.cold.338","s+":"2E"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B74116E","b":"55E949987000","o":"1DBA16E","s":"_ZNK5mongo17BtreeKeyGenerator20_getKeysWithoutArrayERKNS_7BSONObjEN5boost8optionalINS_8RecordIdEEEPSt3setINS_9KeyString5ValueESt4lessISA_ESaISA_EE","s+":"1CE"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B742E2F","b":"55E949987000","o":"1DBBE2F","s":"_ZNK5mongo17BtreeKeyGenerator7getKeysERKNS_7BSONObjEbPSt3setINS_9KeyString5ValueESt4lessIS6_ESaIS6_EEPSt6vectorIS4_ImS7_ImESaImEESaISF_EEN5boost8optionalINS_8RecordIdEEE","s+":"9F"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B383D4E","b":"55E949987000","o":"19FCD4E","s":"_ZNK5mongo17BtreeAccessMethod9doGetKeysERKNS_7BSONObjENS_17IndexAccessMethod14GetKeysContextEPSt3setINS_9KeyString5ValueESt4lessIS8_ESaIS8_EESD_PSt6vectorIS6_ImS9_ImESaImEESaISH_EEN5boost8optionalINS_8RecordIdEEE","s+":"4E"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B459BFA","b":"55E949987000","o":"1AD2BFA","s":"_ZNK5mongo25AbstractIndexAccessMethod7getKeysERKNS_7BSONObjENS_17IndexAccessMethod11GetKeysModeENS4_14GetKeysContextEPSt3setINS_9KeyString5ValueESt4lessIS9_ESaIS9_EESE_PSt6vectorIS7_ImSA_ImESaImEESaISI_EEN5boost8optionalINS_8RecordIdEEESt8functionIFvNS_6StatusES3_SP_EE","s+":"8A"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B3F800B","b":"55E949987000","o":"1A7100B","s":"_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmNS_11unowned_ptrINS_20SeekableRecordCursorEEERKNS_15NamespaceStringE","s+":"7AB"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B3C0166","b":"55E949987000","o":"1A39166","s":"_ZN5mongo10FetchStage6doWorkEPm","s+":"136"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B3D6F14","b":"55E949987000","o":"1A4FF14","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B3D5B1B","b":"55E949987000","o":"1A4EB1B","s":"_ZN5mongo7OrStage6doWorkEPm","s+":"8B"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B3D6F14","b":"55E949987000","o":"1A4FF14","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B3C0089","b":"55E949987000","o":"1A39089","s":"_ZN5mongo10FetchStage6doWorkEPm","s+":"59"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B3D6F14","b":"55E949987000","o":"1A4FF14","s":"_ZN5mongo9PlanStage4workEPm","s+":"64"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B3D0196","b":"55E949987000","o":"1A49196","s":"_ZN5mongo14MultiPlanStage12workAllPlansEmPNS_15PlanYieldPolicyE","s+":"136"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B3D0857","b":"55E949987000","o":"1A49857","s":"_ZN5mongo14MultiPlanStage12pickBestPlanEPNS_15PlanYieldPolicyE","s+":"A7"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B41E302","b":"55E949987000","o":"1A97302","s":"_ZN5mongo16PlanExecutorImpl13_pickBestPlanEv","s+":"92"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B42099A","b":"55E949987000","o":"1A9999A","s":"_ZN5mongo16PlanExecutorImpl4makeEPNS_16OperationContextESt10unique_ptrINS_10WorkingSetESt14default_deleteIS4_EES3_INS_9PlanStageES5_IS8_EES3_INS_13QuerySolutionES5_ISB_EES3_INS_14CanonicalQueryES5_ISE_EERKN5boost13intrusive_ptrINS_17ExpressionContextEEEPKNS_10CollectionENS_15NamespaceStringENS_12PlanExecutor11YieldPolicyE","s+":"18A"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B4211BC","b":"55E949987000","o":"1A9A1BC","s":"_ZN5mongo12PlanExecutor4makeESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS2_EES1_INS_10WorkingSetES3_IS6_EES1_INS_9PlanStageES3_IS9_EEPKNS_10CollectionENS0_11YieldPolicyENS_15NamespaceStringES1_INS_13QuerySolutionES3_ISH_EE","s+":"10C"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B415131","b":"55E949987000","o":"1A8E131","s":"_ZN5mongo11getExecutorEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EENS_12PlanExecutor11YieldPolicyEm","s+":"151"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B41529F","b":"55E949987000","o":"1A8E29F","s":"_ZN5mongo12_GLOBAL__N_116_getExecutorFindEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS6_EENS_12PlanExecutor11YieldPolicyEm","s+":"6F"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B415331","b":"55E949987000","o":"1A8E331","s":"_ZN5mongo15getExecutorFindEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_14CanonicalQueryESt14default_deleteIS5_EEbm","s+":"51"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B40086C","b":"55E949987000","o":"1A7986C","s":"_ZN5mongo12_GLOBAL__N_120attemptToGetExecutorERKN5boost13intrusive_ptrINS_17ExpressionContextEEEPNS_10CollectionERKNS_15NamespaceStringENS_7BSONObjESC_RKSt6bitsetILm11EESC_NS1_8optionalIxEENSH_INSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEEPKNS_18AggregationRequestEmRKy.isra.821","s+":"73C"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B400F3B","b":"55E949987000","o":"1A79F3B","s":"_ZN5mongo9PipelineD15prepareExecutorERKN5boost13intrusive_ptrINS_17ExpressionContextEEEPNS_10CollectionERKNS_15NamespaceStringEPNS_8PipelineERKNS2_INS_18DocumentSourceSortEEESt10unique_ptrINS_36GroupFromFirstDocumentTransformationESt14default_deleteISJ_EESt6bitsetILm11EERKNS_7BSONObjENS1_8optionalIxEEPKNS_18AggregationRequestERKyPb","s+":"55B"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B4014F5","b":"55E949987000","o":"1A7A4F5","s":"_ZN5mongo9PipelineD30buildInnerQueryExecutorGenericEPNS_10CollectionERKNS_15NamespaceStringEPKNS_18AggregationRequestEPNS_8PipelineE","s+":"235"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B402C8A","b":"55E949987000","o":"1A7BC8A","s":"_ZN5mongo9PipelineD23buildInnerQueryExecutorEPNS_10CollectionERKNS_15NamespaceStringEPKNS_18AggregationRequestEPNS_8PipelineE","s+":"54A"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B0A0FFD","b":"55E949987000","o":"1719FFD","s":"_ZN5mongo12runAggregateEPNS_16OperationContextERKNS_15NamespaceStringERKNS_18AggregationRequestERKNS_18LiteParsedPipelineERKNS_7BSONObjERKSt6vectorINS_9PrivilegeESaISF_EEPNS_3rpc21ReplyBuilderInterfaceE","s+":"284D"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B0987BC","b":"55E949987000","o":"17117BC","s":"_ZN5mongo12_GLOBAL__N_115PipelineCommand10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","s+":"6C"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94B66985F","b":"55E949987000","o":"1CE285F","s":"_ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","s+":"7F"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADBBBEF","b":"55E949987000","o":"1434BEF","s":"_ZN5mongo12_GLOBAL__N_114runCommandImplEPNS_16OperationContextEPNS_17CommandInvocationERKNS_12OpMsgRequestEPNS_3rpc21ReplyBuilderInterfaceENS_11LogicalTimeERKNS_23ServiceEntryPointCommon5HooksEPNS_14BSONObjBuilderERKNS_30OperationSessionInfoFromClientE","s+":"9FF"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADBEC3E","b":"55E949987000","o":"1437C3E","s":"_ZZN5mongo12_GLOBAL__N_116receivedCommandsEPNS_16OperationContextERKNS_7MessageERKNS_23ServiceEntryPointCommon5HooksEENKUlvE_clEv","s+":"1C9E"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADBFE7D","b":"55E949987000","o":"1438E7D","s":"_ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageERKNS0_5HooksE","s+":"51D"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADADACC","b":"55E949987000","o":"1426ACC","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","s+":"3C"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADB8A32","b":"55E949987000","o":"1431A32","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE","s+":"F2"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADB6775","b":"55E949987000","o":"142F775","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"125"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADB7606","b":"55E949987000","o":"1430606","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94C0D584A","b":"55E949987000","o":"274E84A","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE","s+":"18A"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADB5A7B","b":"55E949987000","o":"142EA7B","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE","s+":"DB"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADB6D4D","b":"55E949987000","o":"142FD4D","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE","s+":"50D"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADB6E30","b":"55E949987000","o":"142FE30","s":"_ZN5mongo14future_details4callIRZNS_19ServiceStateMachine14_sourceMessageENS2_11ThreadGuardEEUlNS_10StatusWithINS_7MessageEEEE0_S6_EEDaOT_OT0_","s+":"60"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADB70D5","b":"55E949987000","o":"14300D5","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE","s+":"145"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADB67EA","b":"55E949987000","o":"142F7EA","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE","s+":"19A"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94ADB7606","b":"55E949987000","o":"1430606","s":"_ZNSt17_Function_handlerIFvvEZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS2_11ThreadGuardENS1_9transport15ServiceExecutor13ScheduleFlagsENS4_23ServiceExecutorTaskNameENS2_9OwnershipEEUlvE_E9_M_invokeERKSt9_Any_data","s+":"56"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94C0D5EC4","b":"55E949987000","o":"274EEC4","s":"_ZNSt17_Function_handlerIFvvEZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIS0_ENS2_15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameEEUlvE0_E9_M_invokeERKSt9_Any_data","s+":"D4"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94C426F06","b":"55E949987000","o":"2A9FF06","s":"_ZNSt17_Function_handlerIFvvEZN5mongo25launchServiceWorkerThreadESt8functionIS0_EEUlvE1_E9_M_invokeERKSt9_Any_data","s+":"56"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"55E94C426F74","b":"55E949987000","o":"2A9FF74","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","s+":"14"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"7F1D082736DB","b":"7F1D0826C000","o":"76DB","s":"start_thread","s+":"DB"}}}
{"t":{"$date":"2021-06-28T04:10:07.802+00:00"},"s":"I", "c":"CONTROL", "id":31427, "ctx":"conn11630","msg":" Frame: {frame}","attr":{"frame":{"a":"7F1D07F9C71F","b":"7F1D07E7B000","o":"12171F","s":"clone","s+":"3F"}}}



 Comments   
Comment by Mihai Andrei [ 16/Dec/21 ]

Thanks for the reply recom01@gmail.com! Given that this investigation has gone cold, I'm going to close it for the time being as "Cannot Reproduce". If this crash ever reproduces in the future, don't hesitate to re-open this ticket.

Regards,
Mihai

Comment by Viktor Smirnov [ 09/Dec/21 ]

Thank you for your time @Mihai Andrei!

I'm afraid it's not viable for us at the moment to investigate this issue further, we don't keep the logs so much time, and sure we have a lot of requests with $match, $or. Anyway we will bear in mind your suggestions if we bump into this issue in the future.

Comment by Mihai Andrei [ 29/Nov/21 ]

Hey recom01@gmail.com,

Unfortunately, we cannot directly pinpoint the collection where the invariant fired from the stack trace alone. Additionally, I don’t think running validate on every collection will help a lot because we would still need to identify both the collection and the query that caused this invariant to fire (this might be tricky to do if there are 100s of collections, as validate might only be useful if we know the failing collection; knowing the collection would allow us to confirm whether there's some issue with the underlying index or whether it's simply the query shape that's responsible). As such, there are a couple of other things that we can do to try and narrow down the collection/query that caused this invariant:

  • One idea is that we can inspect the logs on each server that crashed from before/after the time when the invariant fired (i.e. around 2021-06-28T04:10:07.802+00:00 that is, on June 28th of this year) as this might give us some idea as to which collection/query caused the invariant to fire (i.e. we can try to match the connection shown in each stack trace to the logs to gather more information). If you still have the logs for both nodes that crashed from around June 28th handy, would you mind providing them? dmitry.agranat should be able to provide a secure uploader that will let you upload the logs.
  • If you’re unable to provide the logs, another idea is that we can narrow down the set of possible collections/queries based on the stack trace. In particular, we can infer from both stack traces that the failing query is an aggregate with a $match which features an $or (something like:

    aggregate([{$match: {$or: ….}}])


    ). If at all possible, would you be able to figure out which queries with that shape are being run in your cluster and/or against which collections? I recognize this might not be easy as a $match + $or query is a common query shape.

Let me know if you have any questions about any of the above!

Thanks,
Mihai

Comment by Viktor Smirnov [ 10/Nov/21 ]

Thank you for the reply Mihai Andrei!

Could we possibly pinpoint the collection and db from the stack trace? Otherwise it requires to run validate() on all the collections and we have hundreds of them across dozens of databases. And the cumulative size is about a couple hundreds GB.

If not, maybe there's a method to batch run it on all the db's on a server?

Comment by Mihai Andrei [ 10/Nov/21 ]

Hi recom01@gmail.com,

I’m glad to hear that you’re no longer facing the issue! To clarify, I don’t believe that this is an issue when the indexes are created initially (i.e during initial sync). This is because from the stack trace provided, it appears that the invariant fires when an ‘aggregate’ is being run. This makes sense, because during query execution, certain plans will generate index keys internally to verify the integrity of results (this is necessary because of yielding: if a query executes, releases its locks, and then reacquires them some time later, the state of the collection could have changed). Given that, the output from ‘validate’ and/or the shape of the ‘query’ that caused the invariant to fire would help narrow down why the crash happened (I understand that this might not be easy information to get given that the crash happened months ago).

Regards,
Mihai

Comment by Viktor Smirnov [ 10/Nov/21 ]

Hello @Mihai Andrei !

Long time has passed and we didn't have the issue anymore. Wouldn't it crash if we had the mentioned problem with a key every time the indexes are generated? I guess it should happen when we have full sync of a new replica set member?

Comment by Mihai Andrei [ 09/Nov/21 ]

Hi recom01@gmail.com,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please run validate on the collection where the invariant was tripped and provide the output, or provide the query that caused the invariant to fire?

Thanks,
Mihai

Comment by Mihai Andrei [ 19/Oct/21 ]

Hi recom01@gmail.com,

Sorry for the delayed reply! After some investigation, I have some ideas as to what might be causing the invariant to fail, but need some more information to figure out what’s going on:

  • The invariant that’s failing is asserting that none of the data that we are generating index keys for contains an array because the index being used is not multikey. As such, it would be good to verify that the indexes are all correct. Could you run validate on the collection where the invariant is being hit to see if there are any problems with the indexes on the collection
  • Similarly, do you know which query is causing the invariant to trip? It would be good to determine whether a particular query shape/index combination is causing this invariant to fail (i.e. to rule out a similar case to SERVER-52589 where a certain index/query shape combination trips the invariant because we’re storing arrays, but the index is not multikey).

Let me know if you have any questions!

Comment by Dmitry Agranat [ 16/Aug/21 ]

Thanks yan.mosyagin@sandsiv.com, I can confirm this is the same issue. Please follow this ticket for updates.

Comment by Yan Mosyagin [ 16/Aug/21 ]

Hi. We're seeing a similar issue after upgrading MongoDB 3.4 to 4.4.8 (standalone) running on an up-to-date Ubuntu 20.04.2. Crash log is attached: mongodb_4.4.8_ubuntu_20.04.2_invariant_failure.log

Thanks.

Comment by Dmitry Agranat [ 29/Jun/21 ]

Thank you recom01@gmail.com for the report. We're assigning this ticket to the appropriate team to be evaluated against our currently planned work. Updates will be posted on this ticket as they happen.

Thanks,
Dima

Comment by Viktor Smirnov [ 28/Jun/21 ]

Backtrace for server 2 is in the attachment.

Generated at Thu Feb 08 05:43:37 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.