|
I was able to confirm that this is indeed a real bug in SBE. If an SBE cursor is idle during a rollback, and then the client issues a getMore command against it, the getMore will currently crash the server while attempting to restore a now-bogus storage cursor:
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0DD0B1E6","b":"7EFF0DB12000","o":"1F91E6","s":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*)","s ":"126"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0DD0C357","b":"7EFF0DB12000","o":"1FA357","s":"mongo::printStackTrace()","s ":"27"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0DD07B1B","b":"7EFF0DB12000","o":"1F5B1B","s":"abruptQuitWithAddrSignal","s ":"18B"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0BE00FD0","b":"7EFF0BDC2000","o":"3EFD0","s":"killpg","s ":"40"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF04B1636F","b":"7EFF04A52000","o":"C436F","s":"mongo::WiredTigerSession::getNewCursor(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, char const*)","s ":"30F"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF04AB1762","b":"7EFF04A52000","o":"5F762","s":"mongo::WiredTigerCursor::WiredTigerCursor(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, bool, mongo::OperationContext*)","s ":"242"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF04AFE95C","b":"7EFF04A52000","o":"AC95C","s":"mongo::WiredTigerRecordStoreCursorBase::restore()","s ":"7C"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF080C6649","b":"7EFF080A9000","o":"1D649","s":"mongo::sbe::ScanStage::doRestoreState()","s ":"69"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF02EBF6D7","b":"7EFF02DFB000","o":"C46D7","s":"mongo::(anonymous namespace)::GetMoreCmd::Invocation::acquireLocksAndIterateCursor(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*, mongo::CursorManager*, mongo::ClientCursorPin&, mongo::CurOp*)","s ":"1367"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF02EBD922","b":"7EFF02DFB000","o":"C2922","s":"mongo::(anonymous namespace)::GetMoreCmd::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)","s ":"332"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF06E9D4E0","b":"7EFF06E68000","o":"354E0","s":"mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)","s ":"80"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF06E9BD26","b":"7EFF06E68000","o":"33D26","s":"mongo::CommandHelpers::runCommandInvocation(std::shared_ptr<mongo::RequestExecutionContext>, std::shared_ptr<mongo::CommandInvocation>, mongo::transport::ServiceExecutor::ThreadingModel)","s ":"1F6"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF05144797","b":"7EFF05109000","o":"3B797","s":"mongo::(anonymous namespace)::runCommandInvocation(std::shared_ptr<mongo::RequestExecutionContext>, std::shared_ptr<mongo::CommandInvocation>)","s ":"B7"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF05140626","b":"7EFF05109000","o":"37626","s":"mongo::(anonymous namespace)::RunCommandImpl::_runCommand()","s ":"366"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0513EEFD","b":"7EFF05109000","o":"35EFD","s":"mongo::(anonymous namespace)::RunCommandAndWaitForWriteConcern::_runImpl()","s ":"5CD"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF051486E0","b":"7EFF05109000","o":"3F6E0","s":"mongo::(anonymous namespace)::RunCommandImpl::run()","s ":"220"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0513B00A","b":"7EFF05109000","o":"3200A","s":"mongo::(anonymous namespace)::ExecCommandDatabase::_commandExec()","s ":"43A"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0513A000","b":"7EFF05109000","o":"31000","s":"mongo::Future<mongo::future_details::UnwrappedTypeImpl<std::invoke_result<mongo::(anonymous namespace)::ExecCommandDatabase::run()::{lambda()#1}&&>::type>::type> mongo::makeReadyFutureWith<mongo::(anonymous namespace)::ExecCommandDatabase::run()::{lambda()#1}, 0>(mongo::(anonymous namespace)::ExecCommandDatabase::run()::{lambda()#1})","s ":"1A40"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF05136EAF","b":"7EFF05109000","o":"2DEAF","s":"_ZN5mongo14future_details12throwingCallIRZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINS2_13HandleRequest16ExecutionContextEEE4$_24JNS0_8FakeVoidEEEEDaOT_DpOT0_","s ":"54F"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0514EE4E","b":"7EFF05109000","o":"45E4E","s":"_ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEE4$_24EEDaOT_ENKUlvE_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","s ":"AE"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0B1680D9","b":"7EFF0B0E6000","o":"820D9","s":"mongo::future_details::SharedStateBase::transitionToFinished()","s ":"49"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0B174611","b":"7EFF0B0E6000","o":"8E611","s":"auto mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::generalImpl<mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>*) &&::{lambda(mongo::future_details::FakeVoid&&)#1}, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>*) &&::{lambda(mongo::Status&&)#1}, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>*) &&::{lambda()#1}>(mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>*) &&::{lambda(mongo::future_details::FakeVoid&&)#1}&&, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>*) &&::{lambda(mongo::Status&&)#1}&&, mongo::future_details::FutureImpl<mongo::future_details::FakeVoid>::propagateResultTo(mongo::future_details::SharedStateImpl<mongo::future_details::FakeVoid>*) &&::{lambda()#1}&&)","s ":"41"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF051367CA","b":"7EFF05109000","o":"2D7CA","s":"_ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL__N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEE4$_21EEDaOT_ENKUlvE_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","s ":"DA"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0B1680D9","b":"7EFF0B0E6000","o":"820D9","s":"mongo::future_details::SharedStateBase::transitionToFinished()","s ":"49"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF05135117","b":"7EFF05109000","o":"2C117","s":"_ZN5mongo14future_details12throwingCallIRZNS_12_GLOBAL__N_116receivedCommandsESt10shared_ptrINS2_13HandleRequest16ExecutionContextEEE4$_26JNS0_8FakeVoidEEEEDaOT_DpOT0_","s ":"A17"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF05134519","b":"7EFF05109000","o":"2B519","s":"_ZZNO5mongo14future_details10FutureImplINS0_8FakeVoidEE4thenIZNS_12_GLOBAL__N_116receivedCommandsESt10shared_ptrINS5_13HandleRequest16ExecutionContextEEE4$_26EEDaOT_ENKUlOS2_E_clESD_","s ":"29"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF05132B97","b":"7EFF05109000","o":"29B97","s":"mongo::(anonymous namespace)::CommandOpRunner::run()","s ":"307"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF05130D76","b":"7EFF05109000","o":"27D76","s":"_ZN5mongo14future_details12throwingCallIRZNS_23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageESt10unique_ptrIKNS2_5HooksESt14default_deleteISA_EEE4$_32JNS0_8FakeVoidEEEEDaOT_DpOT0_","s ":"466"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF05130729","b":"7EFF05109000","o":"27729","s":"_ZZNO5mongo14future_details10FutureImplINS0_8FakeVoidEE4thenIZNS_23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageESt10unique_ptrIKNS5_5HooksESt14default_deleteISD_EEE4$_32EEDaOT_ENKUlOS2_E_clESK_","s ":"29"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0512EC28","b":"7EFF05109000","o":"25C28","s":"mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, std::unique_ptr<mongo::ServiceEntryPointCommon::Hooks const, std::default_delete<mongo::ServiceEntryPointCommon::Hooks const> >)","s ":"3A8"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0A63A1B1","b":"7EFF0A630000","o":"A1B1","s":"mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)","s ":"51"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0A6166C8","b":"7EFF0A5E5000","o":"316C8","s":"mongo::transport::ServiceStateMachine::Impl::processMessage()","s ":"258"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0A61B76C","b":"7EFF0A5E5000","o":"3676C","s":"_ZZNO5mongo14future_details10FutureImplINS0_8FakeVoidEE4thenIZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusEE3$_9EEDaOT_ENKUlOS2_E_clESE_","s ":"2C"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0A61703A","b":"7EFF0A5E5000","o":"3203A","s":"mongo::transport::ServiceStateMachine::Impl::startNewLoop(mongo::Status const&)","s ":"12A"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0A61CC2E","b":"7EFF0A5E5000","o":"37C2E","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZZNS_9transport19ServiceStateMachine4Impl12startNewLoopERKS1_ENK4$_11clES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","s ":"6E"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF06F309EF","b":"7EFF06F00000","o":"309EF","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport26ServiceExecutorSynchronous18runOnDataAvailableERKSt10shared_ptrINS5_7SessionEES3_E3$_5EEDaOT_EN12SpecificImpl4callEOS1_","s ":"3F"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF06F2E508","b":"7EFF06F00000","o":"2E508","s":"mongo::transport::ServiceExecutor::schedule(mongo::unique_function<void (mongo::Status)>)::{lambda()#1}::operator()()","s ":"38"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF06F30656","b":"7EFF06F00000","o":"30656","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEE3$_4EEDaOT_EN12SpecificImpl4callEv","s ":"1D6"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF06F320FE","b":"7EFF06F00000","o":"320FE","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_E3$_3EEDaOT_EN12SpecificImpl4callEv","s ":"5E"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF06F31C78","b":"7EFF06F00000","o":"31C78","s":"mongo::(anonymous namespace)::runFunc(void*)","s ":"18"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0B9A26DB","b":"7EFF0B99B000","o":"76DB","s":"start_thread","s ":"DB"}}}
|
[js_test:read_operations_during_rollback] d20030| {"t":{"$date":"2021-03-24T16:18:19.742-04:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn92","msg":"Frame","attr":{"frame":{"a":"7EFF0BEE3A3F","b":"7EFF0BDC2000","o":"121A3F","s":"clone","s ":"3F"}}}
|
So this fix is very much necessary even though the reopenCatalog command is a thing of the past. I will add a test for the aforementioned rollback scenario, although it will not be tested with SBE until SERVER-55009 is done.
|