Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-26270

Deadlock on Windows when calling currentOp() with bad $where

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.4.0-rc3
    • Component/s: None
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Operating System:
      Windows
    • Sprint:
      Platforms 2016-10-31, Platforms 2016-11-21
    • Linked BF Score:
      0

      Description

      1. The MozJSProxyScope thread (id=18fc) is waiting on the MozJSImplScope thread (id=1e14). It is holding the ServiceContext::_mutex while doing so because CurrentOpCommand::run() runs WhereMatchExpression::matches() on the information extracted from each client as it iterates through all of them.
      2. The MozJSImplScope thread (id=1e14) is waiting to acquire the LdrpLoaderLock critical section and print a stack trace.
      3. The ClientDeleter thread (id=f68) is waiting to acquire the ServiceContext::_mutex and is holding the LdrpLoaderLock critical section while waiting to do so.

      . 21  Id: 19b8.18fc Suspend: 1 Teb: 000007ff`fff90000 Unfrozen
            Start: ucrtbase!crt_at_quick_exit+0x20 (000007fe`ecd9bdc0)
            Priority: 0  Priority class: 32  Affinity: f
      Child-SP          RetAddr           Call Site
      00000000`0875cd18 00000000`77880edd ntdll!ZwWaitForKeyedEvent+0xa
      00000000`0875cd20 00000000`776eb618 ntdll!RtlSleepConditionVariableSRW+0xfd
      00000000`0875cd90 000007fe`ee9187d8 kernel32!SleepConditionVariableSRW+0x28
      (Inline Function) --------`-------- MSVCP140!Concurrency::details::stl_condition_variable_win7::wait_for+0x14 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 216]
      00000000`0875cdd0 000007fe`ee9189a0 MSVCP140!Concurrency::details::stl_condition_variable_win7::wait+0x18 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 210]
      00000000`0875ce00 00000001`3fc65590 MSVCP140!do_wait+0x4c [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 77]
      (Inline Function) --------`-------- mongod!std::_Cnd_waitX+0x10 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95]
      (Inline Function) --------`-------- mongod!std::condition_variable::wait+0x10 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 565]
      (Inline Function) --------`-------- mongod!std::condition_variable::wait+0x26 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 572]
      00000000`0875ce80 00000001`3fc636cf mongod!mongo::mozjs::MozJSProxyScope::runOnImplThread+0xe0 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\scripting\mozjs\proxyscope.cpp @ 281]
      00000000`0875cf10 00000001`3fc65270 mongod!mongo::mozjs::MozJSProxyScope::run<<lambda_ad6871c15ed96f960afd98097b625353> >+0xcf [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\scripting\mozjs\proxyscope.cpp @ 270]
      00000000`0875cff0 00000001`3f7efa45 mongod!mongo::mozjs::MozJSProxyScope::invoke+0x60 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\scripting\mozjs\proxyscope.cpp @ 213]
      00000000`0875d070 00000001`3f7d691e mongod!mongo::WhereMatchExpression::matches+0xf5 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\matcher\expression_where.cpp @ 100]
      00000000`0875d240 00000001`3f698ce8 mongod!mongo::MatchExpression::matchesBSON+0x4e [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\matcher\expression.cpp @ 55]
      00000000`0875d380 00000001`3f70a498 mongod!mongo::CurrentOpCommand::run+0x6f8 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\commands\current_op.cpp @ 151]
      00000000`0875dd00 00000001`3f705705 mongod!mongo::Command::run+0xba8 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\dbcommands.cpp @ 1569]
      00000000`0875e3e0 00000001`3fa5be35 mongod!mongo::Command::execCommand+0x855 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\dbcommands.cpp @ 1411]
      00000000`0875e890 00000001`3f7d37df mongod!mongo::runCommands+0x4c5 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\run_commands.cpp @ 72]
      00000000`0875eb40 00000001`3f7d028c mongod!mongo::`anonymous namespace'::receivedRpc+0x19f [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\instance.cpp @ 273]
      00000000`0875ecd0 00000001`402a46d4 mongod!mongo::assembleResponse+0x7dc [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\instance.cpp @ 617]
      00000000`0875f4a0 00000001`3fc8fc49 mongod!mongo::MyMessageHandler::process+0xb4 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\db.cpp @ 188]
      00000000`0875f660 00000001`3fc8f505 mongod!mongo::PortMessageServer::handleIncomingMsg+0x439 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\net\message_server_port.cpp @ 212]
      (Inline Function) --------`-------- mongod!std::_Invoker_functor::_Call+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
      (Inline Function) --------`-------- mongod!std::invoke+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
      (Inline Function) --------`-------- mongod!std::_Invoke_ret+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1498]
      (Inline Function) --------`-------- mongod!std::_Call_binder+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 829]
      (Inline Function) --------`-------- mongod!std::_Binder<std::_Unforced,void * (__cdecl*)(void *),std::pair<mongo::AbstractMessagingPort *,std::shared_ptr<mongo::MessageHandler> > *>::operator()+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 885]
      (Inline Function) --------`-------- mongod!std::_Invoker_functor::_Call+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
      (Inline Function) --------`-------- mongod!std::invoke+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
      (Inline Function) --------`-------- mongod!std::_LaunchPad<std::unique_ptr<std::tuple<std::_Binder<std::_Unforced,void * (__cdecl*)(void *),std::pair<mongo::AbstractMessagingPort *,std::shared_ptr<mongo::MessageHandler> > *> >,std::default_delete<std::tuple<std::_Binder<std::_Unforced,void * (__cdecl*)(void *),std::pair<mongo::AbstractMessagingPort *,std::shared_ptr<mongo::MessageHandler> > *> > > > >::_Execute+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240]
      00000000`0875fa10 00000001`3f5f43c9 mongod!std::_LaunchPad<std::unique_ptr<std::tuple<std::_Binder<std::_Unforced,void * __ptr64 (__cdecl*)(void * __ptr64),std::pair<mongo::AbstractMessagingPort * __ptr64,std::shared_ptr<mongo::MessageHandler> > * __ptr64> >,std::default_delete<std::tuple<std::_Binder<std::_Unforced,void * __ptr64 (__cdecl*)(void * __ptr64),std::pair<mongo::AbstractMessagingPort * __ptr64,std::shared_ptr<mongo::MessageHandler> > * __ptr64> > > > >::_Run+0x75 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247]
      00000000`0875fa50 000007fe`ecd9be1d mongod!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210]
      00000000`0875fa80 00000000`776b59bd ucrtbase!crt_at_quick_exit+0x7d
      00000000`0875fab0 00000000`777ea2e1 kernel32!BaseThreadInitThunk+0xd
      00000000`0875fae0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
       
      . 22  Id: 19b8.f68 Suspend: 1 Teb: 000007ff`fff8e000 Unfrozen
            Start: ucrtbase!crt_at_quick_exit+0x20 (000007fe`ecd9bdc0)
            Priority: 0  Priority class: 32  Affinity: f
      Child-SP          RetAddr           Call Site
      00000000`0887f518 00000000`777c7c56 ntdll!ZwWaitForKeyedEvent+0xa
      00000000`0887f520 000007fe`ee918da5 ntdll!RtlAcquireSRWLockExclusive+0x106
      00000000`0887f580 00000001`3faa8bb5 MSVCP140!mtx_do_lock+0x91 [f:\dd\vctools\crt\crtw32\stdcpp\thr\mutex.c @ 100]
      (Inline Function) --------`-------- mongod!std::_Mtx_lockX+0xa [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 70]
      (Inline Function) --------`-------- mongod!std::_Mutex_base::lock+0xa [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 43]
      (Inline Function) --------`-------- mongod!std::lock_guard<std::mutex>::{ctor}+0xa [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 218]
      00000000`0887f5e0 00000001`3f67de26 mongod!mongo::ServiceContext::ClientDeleter::operator()+0x35 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\db\service_context.cpp @ 180]
      (Inline Function) --------`-------- mongod!std::unique_ptr<mongo::Client,mongo::ServiceContext::ClientDeleter>::{dtor}+0x10 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\memory @ 1397]
      00000000`0887f650 00000001`3fd4447f mongod!boost::thread_specific_ptr<std::unique_ptr<mongo::Client,mongo::ServiceContext::ClientDeleter> >::delete_data::operator()+0x26 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\third_party\boost-1.60.0\boost\thread\tss.hpp @ 42]
      00000000`0887f690 00000000`7781616e mongod!boost::`anonymous namespace'::run_thread_exit_callbacks+0xdf [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\third_party\boost-1.60.0\libs\thread\src\win32\thread.cpp @ 289]
      00000000`0887f6e0 00000000`7781792d ntdll!LdrpCallTlsInitializers+0xc8
      00000000`0887f750 00000000`77801818 ntdll!LdrShutdownThread+0x1cf
      00000000`0887f850 000007fe`fd573807 ntdll!RtlExitUserThread+0x38
      00000000`0887f890 000007fe`ecd9c031 KERNELBASE!FreeLibraryAndExitThread+0x47
      00000000`0887f8c0 000007fe`ecd9be24 ucrtbase!crt_at_quick_exit+0x291
      00000000`0887f8f0 00000000`776b59bd ucrtbase!crt_at_quick_exit+0x84
      00000000`0887f920 00000000`777ea2e1 kernel32!BaseThreadInitThunk+0xd
      00000000`0887f950 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
       
      . 23  Id: 19b8.1e14 Suspend: 1 Teb: 000007ff`fff8c000 Unfrozen
            Start: ucrtbase!crt_at_quick_exit+0x20 (000007fe`ecd9bdc0)
            Priority: 0  Priority class: 32  Affinity: f
      Child-SP          RetAddr           Call Site
      00000000`082ee508 00000000`77808db8 ntdll!ZwWaitForSingleObject+0xa
      00000000`082ee510 00000000`77808cb4 ntdll!RtlpWaitOnCriticalSection+0xe8
      00000000`082ee5c0 00000000`7787f40f ntdll!RtlEnterCriticalSection+0xd1
      00000000`082ee5f0 00000000`778678ad ntdll!LdrQueryModuleInfoLocalLoaderLock+0x1f
      00000000`082ee620 00000000`7788d78d ntdll!LdrQueryProcessModuleInformationEx+0x8d
      00000000`082ee6d0 00000000`778a75d1 ntdll!RtlQueryProcessModuleInformation+0x4d
      00000000`082ee720 000007fe`f9d9201e ntdll!RtlQueryProcessDebugInformation+0x371
      00000000`082ee8b0 000007fe`f9d84756 dbghelp!GetProcessModules+0x6e
      00000000`082ee900 000007fe`f9d847da dbghelp!SymInitializeW+0x202
      00000000`082eeb70 00000001`3fcd190e dbghelp!SymInitialize+0x2e
      00000000`082eeba0 00000001`3fcd1870 mongod!mongo::printWindowsStackTrace+0x7e [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\stacktrace_windows.cpp @ 189]
      00000000`082eef90 00000001`3fcd0a6d mongod!mongo::printStackTrace+0x50 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\stacktrace_windows.cpp @ 175]
      00000000`082ef4a0 00000001`3fc6d780 mongod!mongo::printStackTrace+0x4d [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\stacktrace.cpp @ 43]
      00000000`082ef560 00000001`3fc6d9c1 mongod!mongo::DBException::traceIfNeeded+0xe0 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\assert_util.cpp @ 90]
      (Inline Function) --------`-------- mongod!mongo::DBException::{ctor}+0x55 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\assert_util.h @ 93]
      (Inline Function) --------`-------- mongod!mongo::AssertionException::{ctor}+0x55 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\assert_util.h @ 141]
      (Inline Function) --------`-------- mongod!mongo::UserException::{ctor}+0x55 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\assert_util.h @ 156]
      00000000`082ef640 00000001`3fc6d7c3 mongod!mongo::uasserted+0x1f1 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\assert_util.cpp @ 211]
      00000000`082ef730 00000001`3fc49019 mongod!mongo::uasserted+0x13 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\assert_util.cpp @ 198]
      (Inline Function) --------`-------- mongod!mongo::uassertStatusOK::__l7::<lambda_b4227dbd1dc5832f71dd04874ced1b2e>::operator()+0x23 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\assert_util.h @ 247]
      (Inline Function) --------`-------- mongod!mongo::uassertStatusOK+0x28 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\util\assert_util.h @ 247]
      00000000`082ef760 00000001`3fc4fc50 mongod!mongo::mozjs::MozJSImplScope::_checkErrorState+0x389 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\scripting\mozjs\implscope.cpp @ 833]
      00000000`082ef990 00000001`3fc63acc mongod!mongo::mozjs::MozJSImplScope::invoke+0x420 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\scripting\mozjs\implscope.cpp @ 623]
      00000000`082efc00 00000001`3fc65035 mongod!<lambda_ad6871c15ed96f960afd98097b625353>::operator()+0x5c [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\scripting\mozjs\proxyscope.cpp @ 209]
      (Inline Function) --------`-------- mongod!std::_Func_class<void>::operator()+0x18 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
      00000000`082efc50 00000001`3fc39c7c mongod!mongo::mozjs::MozJSProxyScope::implThread+0x135 [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\scripting\mozjs\proxyscope.cpp @ 347]
      00000000`082efcc0 00000001`3fc39d25 mongod!nspr::Thread::ThreadRoutine+0x2c [c:\data\mci\f279e5d64c77d9d92759b2c518207fa8\src\src\mongo\scripting\mozjs\posixnspr.cpp @ 58]
      (Inline Function) --------`-------- mongod!std::_Invoker_functor::_Call+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
      (Inline Function) --------`-------- mongod!std::invoke+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
      (Inline Function) --------`-------- mongod!std::_LaunchPad<std::unique_ptr<std::tuple<void * (__cdecl*)(void *),nspr::Thread *>,std::default_delete<std::tuple<void * (__cdecl*)(void *),nspr::Thread *> > > >::_Execute+0x6 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240]
      00000000`082efcf0 00000001`3f5f43c9 mongod!std::_LaunchPad<std::unique_ptr<std::tuple<void * __ptr64 (__cdecl*)(void * __ptr64),nspr::Thread * __ptr64>,std::default_delete<std::tuple<void * __ptr64 (__cdecl*)(void * __ptr64),nspr::Thread * __ptr64> > > >::_Run+0x75 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247]
      00000000`082efd30 000007fe`ecd9be1d mongod!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210]
      00000000`082efd60 00000000`776b59bd ucrtbase!crt_at_quick_exit+0x7d
      00000000`082efd90 00000000`777ea2e1 kernel32!BaseThreadInitThunk+0xd
      00000000`082efdc0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
       
      -----------------------------------------
      DebugInfo          = 0x00000000778f2100
      Critical section   = 0x00000000778f2490 (ntdll!LdrpLoaderLock+0x0)
      LOCKED
      LockCount          = 0x1
      WaiterWoken        = No
      OwningThread       = 0x0000000000000f68
      RecursionCount     = 0x1
      LockSemaphore      = 0x214
      SpinCount          = 0x0000000000000000
      

        Attachments

          Activity

            People

            Assignee:
            jonathan.reams Jonathan Reams
            Reporter:
            max.hirschhorn Max Hirschhorn
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: