Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-21723

Replica set nodes hang when awaiting replication in DBDirectClient

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 2.6.11
    • Component/s: Replication
    • Replication
    • ALL

      This bug affects versions 2.6+.

        thread #65: tid = 0x6907a, 0x00007fff89a1d136 libsystem_kernel.dylib`__psynch_cvwait + 10
          frame #0: 0x00007fff89a1d136 libsystem_kernel.dylib`__psynch_cvwait + 10
          frame #1: 0x00007fff872ac560 libsystem_pthread.dylib`_pthread_cond_wait + 693
          frame #2: 0x00007fff8724fc95 libc++.1.dylib`std::__1::condition_variable::wait(std::__1::unique_lock<std::__1::mutex>&) + 47
          frame #3: 0x000000010a80ddf6 mongod`mongo::repl::ReplicationCoordinatorImpl::_awaitReplication_inlock(this=<unavailable>, timer=<unavailable>, lock=0x0000000112274ff0, txn=<unavailable>, opTime=0x0000000112275190, minSnapshot=<unavailable>, writeConcern=<unavailable>) + 966 at replication_coordinator_impl.cpp:1263
          frame #4: 0x000000010a80d9f4 mongod`mongo::repl::ReplicationCoordinatorImpl::awaitReplication(this=<unavailable>, txn=<unavailable>, opTime=<unavailable>, writeConcern=<unavailable>) + 100 at replication_coordinator_impl.cpp:1171
          frame #5: 0x000000010a9b278d mongod`mongo::waitForWriteConcern(txn=0x00007ff472708160, replOpTime=0x0000000112275190, writeConcern=0x00007ff4727081a0, result=0x00000001122751a8) + 317 at write_concern.cpp:288
          frame #6: 0x000000010a4b5919 mongod`mongo::WriteBatchExecutor::executeBatch(this=0x00000001122754a0, request=0x00000001122755c0, response=0x0000000112275518) + 1049 at batch_executor.cpp:336
          frame #7: 0x000000010a4bf058 mongod`mongo::WriteCmd::run(this=<unavailable>, txn=<unavailable>, dbName=<unavailable>, cmdObj=<unavailable>, options=<unavailable>, errMsg=<unavailable>, result=0x0000000112275878) + 488 at write_commands.cpp:144
          frame #8: 0x000000010a4daf4e mongod`mongo::Command::run(this=<unavailable>, txn=0x00007ff472708160, request=0x0000000112275f00, replyBuilder=0x00000001122760d8) + 2238 at dbcommands.cpp:1432
          frame #9: 0x000000010a4da21b mongod`mongo::Command::execCommand(txn=0x00007ff472708160, command=0x00007ff4726011a0, request=<unavailable>, replyBuilder=0x00000001122760d8) + 3707 at dbcommands.cpp:1317
          frame #10: 0x000000010a42db97 mongod`mongo::runCommands(txn=0x00007ff472708160, request=0x0000000112275f00, replyBuilder=0x00000001122760d8) + 679 at commands.cpp:498
          frame #11: 0x000000010a5d9ef0 mongod`mongo::receivedCommand(txn=<unavailable>, nss=<unavailable>, client=<unavailable>, dbResponse=<unavailable>, message=<unavailable>) + 288 at instance.cpp:252
          frame #12: 0x000000010a5d6051 mongod`mongo::assembleResponse(txn=0x00007ff472708160, m=0x0000000112278848, dbresponse=0x0000000112278618, remote=0x000000010b7aa640) + 2161 at instance.cpp:517
          frame #13: 0x000000010a4e4046 mongod`mongo::DBDirectClient::call(this=<unavailable>, toSend=<unavailable>, response=0x00007ff47243f418, assertOk=<unavailable>, actualServer=<unavailable>) + 230 at dbdirectclient.cpp:134
          frame #14: 0x000000010a4e414f mongod`non-virtual thunk to mongo::DBDirectClient::call(this=<unavailable>, toSend=<unavailable>, response=<unavailable>, assertOk=<unavailable>, actualServer=<unavailable>) + 15 at dbdirectclient.cpp:142
          frame #15: 0x000000010a33ac71 mongod`mongo::DBClientCursor::init(this=0x00007ff47243f410) + 145 at dbclientcursor.cpp:139
          frame #16: 0x000000010a3253c9 mongod`mongo::DBClientBase::query(this=<unavailable>, ns=<unavailable>, query=<unavailable>, nToReturn=<unavailable>, nToSkip=<unavailable>, fieldsToReturn=0x0000000000000000, queryOptions=0, batchSize=0) + 105 at dbclient.cpp:1085
          frame #17: 0x000000010a4e4352 mongod`mongo::DBDirectClient::query(this=<unavailable>, ns=<unavailable>, query=<unavailable>, nToReturn=<unavailable>, nToSkip=<unavailable>, fieldsToReturn=<unavailable>, queryOptions=<unavailable>, batchSize=<unavailable>) + 82 at dbdirectclient.cpp:160
          frame #18: 0x000000010ab57641 mongod`mongo::mozjs::MongoBase::Functions::find::call(cx=0x00007ff472535170, args=<unavailable>) + 929 at mongo.cpp:222
          frame #19: 0x000000010ab5e55f mongod`bool mongo::mozjs::smUtils::wrapConstrainedMethod<mongo::mozjs::MongoBase::Functions::find, true, mongo::mozjs::MongoLocalInfo, mongo::mozjs::MongoExternalInfo>(cx=0x00007ff472535170, argc=7, vp=<unavailable>) + 1247 at wrapconstrainedmethod.h:113
          frame #20: 0x000000010adf3f16 mongod`js::Invoke(JSContext*, JS::CallArgs, js::MaybeConstruct) [inlined] js::CallJSNative(native=0x000000010ab5e080)(JSContext*, unsigned int, JS::Value*), JS::CallArgs const&) + 69 at jscntxtinlines.h:226
          frame #21: 0x000000010adf3ed1 mongod`js::Invoke(cx=<unavailable>, args=<unavailable>, construct=<unavailable>) + 321 at Interpreter.cpp:498
          frame #22: 0x000000010ae0bfd0 mongod`Interpret(cx=0x00007ff472535170, state=0x000000011227a508) + 56400 at Interpreter.cpp:2602
          frame #23: 0x000000010adfe344 mongod`js::RunScript(cx=0x00007ff472535170, state=0x000000011227a508) + 420 at Interpreter.cpp:448
          frame #24: 0x000000010adf41ac mongod`js::Invoke(cx=<unavailable>, args=<unavailable>, construct=<unavailable>) + 1052 at Interpreter.cpp:517
          frame #25: 0x000000010ade284e mongod`js::Invoke(cx=<unavailable>, thisv=<unavailable>, fval=<unavailable>, argc=<unavailable>, argv=<unavailable>, rval=<unavailable>) + 734 at Interpreter.cpp:554
          frame #26: 0x000000010b151bf9 mongod`JS_CallFunctionValue(cx=<unavailable>, obj=<unavailable>, fval=<unavailable>, args=<unavailable>, rval=<unavailable>) + 73 at jsapi.cpp:4216
          frame #27: 0x000000010ab4c404 mongod`mongo::mozjs::MozJSImplScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool) [inlined] JS::Call(cx=<unavailable>, args=<unavailable>) + 12 at jsapi.h:3754
          frame #28: 0x000000010ab4c3f8 mongod`mongo::mozjs::MozJSImplScope::invoke(this=0x00007ff47285cc00, func=<unavailable>, argsObject=<unavailable>, recv=<unavailable>, timeoutMs=0, ignoreReturn=false, readOnlyArgs=<unavailable>, readOnlyRecv=<unavailable>) + 1192 at implscope.cpp:585
          frame #29: 0x000000010ab6fc75 mongod`std::__1::__function::__func<mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24, std::__1::allocator<mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24>, void ()>::operator()() [inlined] mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24::operator()() const + 69 at proxyscope.cpp:211
          frame #30: 0x000000010ab6fc30 mongod`std::__1::__function::__func<mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24, std::__1::allocator<mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24>, void ()>::operator()() [inlined] decltype(std::__1::forward<mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24&>(fp)(std::__1::forward<>(fp0))) std::__1::__invoke<mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24&>(mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24&&&) + 4 at __functional_base:413
          frame #31: 0x000000010ab6fc2c mongod`std::__1::__function::__func<mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24, std::__1::allocator<mongo::mozjs::MozJSProxyScope::invoke(unsigned long long, mongo::BSONObj const*, mongo::BSONObj const*, int, bool, bool, bool)::$_24>, void ()>::operator(this=0x00007ff47254add0)() + 12 at functional:1370
          frame #32: 0x000000010ab6d199 mongod`mongo::mozjs::MozJSProxyScope::implThread(void*) [inlined] std::__1::function<void ()>::operator()() const + 393 at functional:1756
          frame #33: 0x000000010ab6d181 mongod`mongo::mozjs::MozJSProxyScope::implThread(arg=0x00007ff472706980) + 369 at proxyscope.cpp:351
          frame #34: 0x000000010ab384fc mongod`nspr::Thread::ThreadRoutine(arg=0x00007ff472707d80) + 28 at PosixNSPR.cpp:56
          frame #35: 0x000000010ab38a81 mongod`void* std::__1::__thread_proxy<std::__1::tuple<void* (*)(void*), nspr::Thread*> >(void*) [inlined] decltype(__f=<unavailable>)(void*)>(fp)(std::__1::forward<nspr::Thread*>(fp0))) std::__1::__invoke<void* (*)(void*), nspr::Thread*>(void* (*&&)(void*), nspr::Thread*&&) + 97 at __functional_base:413
          frame #36: 0x000000010ab38a78 mongod`void* std::__1::__thread_proxy<std::__1::tuple<void* (*)(void*), nspr::Thread*> >(void*) [inlined] void std::__1::__thread_execute<void* (*)(void*), nspr::Thread*, 1ul>(std::__1::tuple<void* (*)(void*), nspr::Thread*>&, std::__1::__tuple_indices<1ul>) at thread:332
          frame #37: 0x000000010ab38a78 mongod`void* std::__1::__thread_proxy<std::__1::tuple<void* (*)(void*), nspr::Thread*> >(__vp=0x00007ff472707fc0) + 88 at thread:342
          frame #38: 0x00007fff872ac05a libsystem_pthread.dylib`_pthread_body + 131
          frame #39: 0x00007fff872abfd7 libsystem_pthread.dylib`_pthread_start + 176
          frame #40: 0x00007fff872a93ed libsystem_pthread.dylib`thread_start + 13
      

            Assignee:
            backlog-server-repl [DO NOT USE] Backlog - Replication Team
            Reporter:
            kamran.khan Kamran K.
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: