Issue summary
Symptoms & User-visible impact:
Sharded clusters on versions 6.0.7-6.0.11 and 7.0.0-7.0.3 can see mongod process crashes as a result of a use-after-free memory issue in the $listSearchIndexes aggregation stage.
Key Diagnostics:
- Server mongod process may crash with an Invalid Access at address error and Segmentation fault.
- Log file may contain an Invariant failure referring to the file src/mongo/util/future_impl.h, such as:
msg":"Invariant failure","attr":{"expr":"!callback","file":"src/mongo/util/future_impl.h","line":443}}
- Segmentation faults with backtraces including DocumentSourceListSearchIndexes, such as:
_ZN5mongo18stack_trace_detail12_GLOBAL__N_117getStackTraceImplERKNS1_7OptionsE.constprop.205 _ZN5mongo15printStackTraceEv abruptQuitWithAddrSignal _L_unlock_13 __memcpy_ssse3_back _ZN5mongo31DocumentSourceListSearchIndexes9doGetNextEv _ZN5mongo14DocumentSource7getNextEv _ZN5mongo8Pipeline7getNextEv _ZN5mongo20PlanExecutorPipeline11_tryGetNextEv _ZN5mongo20PlanExecutorPipeline8_getNextEv _ZN5mongo20PlanExecutorPipeline15getNextDocumentEPNS_8DocumentEPNS_8RecordIdE _ZN5mongo20PlanExecutorPipeline7getNextEPNS_7BSONObjEPNS_8RecordIdE _ZN5mongo12_GLOBAL__N_110GetMoreCmd10Invocation28acquireLocksAndIterateCursorEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceERNS_15ClientCursorPinEPNS_5CurOpE _ZN5mongo12_GLOBAL__N_110GetMoreCmd10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE _ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE _ZN5mongo14CommandHelpers20runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEENS_9transport15ServiceExecutor14ThreadingModelE _ZN5mongo12_GLOBAL__N_120runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEE _ZN5mongo12_GLOBAL__N_114RunCommandImpl11_runCommandEv _ZN5mongo12_GLOBAL__N_132RunCommandAndWaitForWriteConcern24_runCommandWithFailPointEv _ZN5mongo12_GLOBAL__N_132RunCommandAndWaitForWriteConcern8_runImplEv _ZN5mongo12_GLOBAL__N_114RunCommandImpl3runEv _ZN5mongo12_GLOBAL__N_119ExecCommandDatabase12_commandExecEv _ZZN5mongo12_GLOBAL__N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEEENUlvE0_clEv _ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE0_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_ _ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv _ZN5mongo14future_details10FutureImplINS0_8FakeVoidEE11generalImplIZNOS3_17propagateResultToEPNS0_15SharedStateImplIS2_EEEUlOS2_E_ZNOS3_17propagateResultToES7_EUlONS_6StatusEE0_ZNOS3_17propagateResultToES7_EUlvE1_EEDaOT_OT0_OT1_ _ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_ _ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv _ZN5mongo12_GLOBAL__N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE _ZN5mongo12_GLOBAL__N_116receivedCommandsESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE _ZN5mongo12_GLOBAL__N_115CommandOpRunner3runEv _ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageESt10unique_ptrIKNS0_5HooksESt14default_deleteIS8_EE _ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE _ZN5mongo9transport19ServiceStateMachine4Impl14processMessageEv _ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE _ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_ _ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport26ServiceExecutorSynchronous18runOnDataAvailableERKSt10shared_ptrINS5_7SessionEES3_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_ _ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv _ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv _ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv _ZN5mongo12_GLOBAL__N_17runFuncEPv start_thread clone
- Users who use Compass may have observed an increase in failures as a result of this issue given the 1.40.0 release of Compass which started offering support for Search indexes.
- In some situations, the server mongod process may become deadlocked after failing to process the abort signal due to
SERVER-82459.
Root cause:
A use-after-free error within the $listSearchIndexes aggregation pipeline stage resulted in a data structure pointing to a freed location within Heap memory. This allows for a wide variety of failure scenarios where threads unrelated to the operation which issued the $listSearchIndexes operation could terminate the server process with Invalid Access segmentation faults.
Fix or Remediations:
- Upgrade to 6.0.12 or 7.0.4 when available.
- Reduce or stop the use of the $listSearchIndexes aggregation stage.
- This may require using a pre-1.40.0 version of Compass until updating to a version of MongoDB containing a fix for this issue.
Previous Description
Current copy is unowned and at risk of being destroyed while held
- is related to
-
SERVER-74863 Implement $listSearchIndexes aggregation stage
-
- Closed
-
- related to
-
-
- Closed
-
-
SERVER-82945 Increase sharded search / vectorSearch testing
-
- Backlog
-