Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-37838

stepDown during a getMore followed by an OP_KILL_CURSORS can crash the server

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 4.0.5, 4.1.6
    • Affects Version/s: 4.0.3
    • Component/s: Querying
    • Labels:
      None
    • Fully Compatible
    • ALL
    • v4.0
    • Hide

      While there is an ongoing change stream operation with an active cursor run rs.stepDown() on the primary.

      This occurred while manually testing the C# driver support for change stream recovery after elections. It's possible there is a driver bug... but whether or not there is the server should not abort.

      Show
      While there is an ongoing change stream operation with an active cursor run rs.stepDown() on the primary. This occurred while manually testing the C# driver support for change stream recovery after elections. It's possible there is a driver bug... but whether or not there is the server should not abort.
    • Query 2018-12-03, Query 2018-12-17
    • 33

      After running rs.stepDown() on the primary the primary crashes with the following output at the tail of the log:

      2018-10-30T16:04:18.438-0400 I NETWORK  [conn71] received client metadata from 192.168.0.122:63244 conn71: { driver: { name: "mongo-csharp-driver", version: "0.0.0.0" }, os: { type: "Windows", name: "Microsoft Windows 10.0.17134", architecture: "x86_32", version: "10.0.17134" }, platform: ".NET Framework 4.7.3190.0" }
      2018-10-30T16:04:18.439-0400 F -        [conn71] Invariant failure pin == ErrorCodes::CursorNotFound || pin == ErrorCodes::Unauthorized src\mongo\db\cursor_manager.cpp 186
      2018-10-30T16:04:18.443-0400 F -        [conn71]
      ***aborting after invariant() failure
      2018-10-30T16:04:18.547-0400 F -        [conn71] Got signal: 22 (SIGABRT).
      mongod.exe    ...\src\mongo\util\stacktrace_windows.cpp(244)                                   mongo::printStackTrace+0x67x
      mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(182)                          mongo::`anonymous namespace'::printSignalAndBacktrace+0x116x
      mongod.exe    ...\src\mongo\util\signal_handlers_synchronous.cpp(238)                          mongo::`anonymous namespace'::abruptQuit+0x133x
      ucrtbase.dll                                                                                   raise+0x475x
      ucrtbase.dll                                                                                   abort+0x49x
      mongod.exe    ...\src\mongo\util\assert_util.cpp(104)                                          mongo::invariantFailed+0x410x
      mongod.exe    ...\src\mongo\db\cursor_manager.cpp(186)                                         mongo::`anonymous namespace'::GlobalCursorIdCache::killCursor+0x208x
      mongod.exe    ...\src\mongo\db\cursor_manager.cpp(371)                                         mongo::CursorManager::killCursorGlobalIfAuthorized+0x73x
      mongod.exe    ...\src\mongo\db\service_entry_point_common.cpp(1127)                            mongo::`anonymous namespace'::receivedKillCursors+0x613x
      mongod.exe    ...\src\mongo\db\service_entry_point_common.cpp(1339)                            mongo::ServiceEntryPointCommon::handleRequest+0x1313x
      mongod.exe    ...\src\mongo\db\service_entry_point_mongod.cpp(123)                             mongo::ServiceEntryPointMongod::handleRequest+0x67x
      mongod.exe    ...\src\mongo\transport\service_state_machine.cpp(378)                           mongo::ServiceStateMachine::_processMessage+0x479x
      mongod.exe    ...\src\mongo\transport\service_state_machine.cpp(440)                           mongo::ServiceStateMachine::_runNextInGuard+0x207x
      mongod.exe    ...\src\mongo\transport\service_state_machine.cpp(479)                           <lambda_b23af5efc3b61ab25bff0c3bcd13382b>::operator()+0x109x
      mongod.exe    ...\src\mongo\transport\service_executor_synchronous.cpp(116)                    mongo::transport::ServiceExecutorSynchronous::schedule+0x283x
      mongod.exe    ...\src\mongo\transport\service_state_machine.cpp(483)                           mongo::ServiceStateMachine::_scheduleNextWithGuard+0x236x
      mongod.exe    ...\src\mongo\transport\service_state_machine.cpp(303)                           mongo::ServiceStateMachine::_sourceCallback+0x289x
      mongod.exe    ...\src\mongo\transport\service_state_machine.cpp(259)                           <lambda_5e72c1d9f04187bee457482c0afa7487>::operator()+0x185x
      mongod.exe    ...\src\mongo\util\future.h(788)                                                 mongo::future_details::Future<mongo::Message>::getAsync<<lambda_5e72c1d9f04187bee457482c0afa7487> >+0x103x
      mongod.exe    ...\src\mongo\transport\service_state_machine.cpp(254)                           mongo::ServiceStateMachine::_sourceMessage+0x192x
      mongod.exe    ...\src\mongo\transport\service_state_machine.cpp(436)                           mongo::ServiceStateMachine::_runNextInGuard+0x242x
      mongod.exe    ...\src\mongo\transport\service_state_machine.cpp(479)                           <lambda_b23af5efc3b61ab25bff0c3bcd13382b>::operator()+0x109x
      mongod.exe    ...\src\mongo\transport\service_executor_synchronous.cpp(133)                    <lambda_472996f9e6b00ec91d31b43a6cde81f7>::operator()+0x217x
      mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247)  std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x133x
      mongod.exe    c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210)  std::_Pad::_Call_func+0x9x
      ucrtbase.dll                                                                                   o_ceil+0x78x
      KERNEL32.DLL                                                                                   BaseThreadInitThunk+0x20x
      2018-10-30T16:04:18.553-0400 F CONTROL  [conn71] *** unhandled exception 0x0000000E at 0x00007FFDB0F9A388, terminating
      2018-10-30T16:04:18.553-0400 F CONTROL  [conn71] *** stack trace for unhandled exception:
      2018-10-30T16:04:18.556-0400 I -        [conn71] KERNELBASE.dll                                                                                   RaiseException+0x104x
      mongod.exe      ...\src\mongo\util\signal_handlers_synchronous.cpp(239)                          mongo::`anonymous namespace'::abruptQuit+0x157x
      ucrtbase.dll                                                                                     raise+0x475x
      ucrtbase.dll                                                                                     abort+0x49x
      mongod.exe      ...\src\mongo\util\assert_util.cpp(104)                                          mongo::invariantFailed+0x410x
      mongod.exe      ...\src\mongo\db\cursor_manager.cpp(186)                                         mongo::`anonymous namespace'::GlobalCursorIdCache::killCursor+0x208x
      mongod.exe      ...\src\mongo\db\cursor_manager.cpp(371)                                         mongo::CursorManager::killCursorGlobalIfAuthorized+0x73x
      mongod.exe      ...\src\mongo\db\service_entry_point_common.cpp(1127)                            mongo::`anonymous namespace'::receivedKillCursors+0x613x
      mongod.exe      ...\src\mongo\db\service_entry_point_common.cpp(1339)                            mongo::ServiceEntryPointCommon::handleRequest+0x1313x
      mongod.exe      ...\src\mongo\db\service_entry_point_mongod.cpp(123)                             mongo::ServiceEntryPointMongod::handleRequest+0x67x
      mongod.exe      ...\src\mongo\transport\service_state_machine.cpp(378)                           mongo::ServiceStateMachine::_processMessage+0x479x
      mongod.exe      ...\src\mongo\transport\service_state_machine.cpp(440)                           mongo::ServiceStateMachine::_runNextInGuard+0x207x
      mongod.exe      ...\src\mongo\transport\service_state_machine.cpp(479)                           <lambda_b23af5efc3b61ab25bff0c3bcd13382b>::operator()+0x109x
      mongod.exe      ...\src\mongo\transport\service_executor_synchronous.cpp(116)                    mongo::transport::ServiceExecutorSynchronous::schedule+0x283x
      mongod.exe      ...\src\mongo\transport\service_state_machine.cpp(483)                           mongo::ServiceStateMachine::_scheduleNextWithGuard+0x236x
      mongod.exe      ...\src\mongo\transport\service_state_machine.cpp(303)                           mongo::ServiceStateMachine::_sourceCallback+0x289x
      mongod.exe      ...\src\mongo\transport\service_state_machine.cpp(259)                           <lambda_5e72c1d9f04187bee457482c0afa7487>::operator()+0x185x
      mongod.exe      ...\src\mongo\util\future.h(788)                                                 mongo::future_details::Future<mongo::Message>::getAsync<<lambda_5e72c1d9f04187bee457482c0afa7487> >+0x103x
      mongod.exe      ...\src\mongo\transport\service_state_machine.cpp(254)                           mongo::ServiceStateMachine::_sourceMessage+0x192x
      mongod.exe      ...\src\mongo\transport\service_state_machine.cpp(436)                           mongo::ServiceStateMachine::_runNextInGuard+0x242x
      mongod.exe      ...\src\mongo\transport\service_state_machine.cpp(479)                           <lambda_b23af5efc3b61ab25bff0c3bcd13382b>::operator()+0x109x
      mongod.exe      ...\src\mongo\transport\service_executor_synchronous.cpp(133)                    <lambda_472996f9e6b00ec91d31b43a6cde81f7>::operator()+0x217x
      mongod.exe      c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(247)  std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x133x
      mongod.exe      c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread(210)  std::_Pad::_Call_func+0x9x
      ucrtbase.dll                                                                                     o_ceil+0x78x
      KERNEL32.DLL                                                                                     BaseThreadInitThunk+0x20x
      2018-10-30T16:04:18.561-0400 I CONTROL  [conn71] writing minidump diagnostic file C:\mongodb\v4.2018-10-30T20-04-18.mdmp
      2018-10-30T16:04:18.674-0400 I REPL     [rsBackgroundSync] sync source candidate: Fractal:27017
      2018-10-30T16:04:18.675-0400 I ASIO     [RS] Connecting to Fractal:27017
      2018-10-30T16:04:18.678-0400 F CONTROL  [conn71] *** immediate exit due to unhandled exception
      

            Assignee:
            ian.boros@mongodb.com Ian Boros
            Reporter:
            robert@mongodb.com Robert Stam
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: