Loading...

XML

Word

Printable

JSON

Type: Bug
Resolution: Gone away
Priority: Minor - P4
Fix Version/s: None
Affects Version/s: None
Component/s: None
Labels:
None

As Louis mentions in his initial investigation, all application threads are stuck trying to evict.

They are either in a state where:

We're initialising a cursor and do an eviction check.
We just committed a transaction (so the pages it was using should no longer be pinned) and do an eviction check.

I've pasted the stack traces for both of these thread states below.

I had a closer look at why we get WT_NOTFOUND from evict_page. In evict_page -> evict_get_ref, we have this:

        /* Avoid the LRU lock if no pages are available. */
        if (__evict_queue_empty(cache->evict_current_queue, is_server) &&
            __evict_queue_empty(cache->evict_other_queue, is_server) &&
            (!urgent_ok || __evict_queue_empty(urgent_queue, false))) {
                WT_STAT_CONN_INCR(session, cache_eviction_get_ref_empty);
                return (WT_NOTFOUND);
        }

Each of those evict_queue_empty calls returns true when we the queue's evict_current member is NULL. And all of them are:

(gdb) p *conn->cache->evict_current_queue
$7 = {
...
  evict_queue = 0x7fe20f0bf000,
  evict_current = 0x0,
  evict_candidates = 100,
  evict_entries = 100,
  evict_max = 400
}

(gdb) p *conn->cache->evict_other_queue
$8 = {
...
  evict_queue = 0x7fe20c65f800,
  evict_current = 0x0,
  evict_candidates = 100,
  evict_entries = 100,
  evict_max = 400
}

(gdb) p *conn->cache->evict_urgent_queue
$9 = {
...
  evict_queue = 0x7fe20f0c1800,
  evict_current = 0x0,
  evict_candidates = 1,
  evict_entries = 0,
  evict_max = 54
}

So I believe that's the code path that is being exercised. I imagine that to understand why it gets to this state, it may be necessary to repro and attach GDB. This page might come in handy: https://wiki.corp.mongodb.com/display/KERNEL/Hang+Analysis.

At the time of the core dump, I don't think there were any transactions in progress so I don't see a good reason why the cache should be pinned and we should be failing to evict.

Eviction during cursor initialisation:

[2019/04/28 13:53:10.442] #0  0x00007fe206224408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 [2019/04/28 13:53:10.442] #1  0x00007fe20885635b in __wt_cond_wait_signal (session=session@entry=0x7fe20cb70900, cond=0x7fe2113a7080, usecs=usecs@entry=10000, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fe1d99ba920) at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:122
 [2019/04/28 13:53:10.442] #2  0x00007fe20884098c in __wt_cond_wait (run_func=0x0, usecs=10000, cond=<optimized out>, session=0x7fe20cb70900) at src/third_party/wiredtiger/src/include/misc.i:19
 [2019/04/28 13:53:10.442] #3  __wt_cache_eviction_worker (session=session@entry=0x7fe20cb70900, busy=<optimized out>, readonly=readonly@entry=false, pct_full=<optimized out>) at src/third_party/wiredtiger/src/evict/evict_lru.c:2424
 [2019/04/28 13:53:10.442] #4  0x00007fe2088b11f3 in __wt_cache_eviction_check (busy=<optimized out>, didworkp=0x0, readonly=false, session=<optimized out>) at src/third_party/wiredtiger/src/include/cache.i:453
 [2019/04/28 13:53:10.442] #5  __cursor_enter (session=<optimized out>) at src/third_party/wiredtiger/src/include/cursor.i:152
 [2019/04/28 13:53:10.442] #6  __cursor_func_init (reenter=true, cbt=0x7fe2116fa300) at src/third_party/wiredtiger/src/include/cursor.i:357
 [2019/04/28 13:53:10.442] #7  __wt_btcur_search (cbt=cbt@entry=0x7fe2116fa300) at src/third_party/wiredtiger/src/btree/bt_cursor.c:591
 [2019/04/28 13:53:10.442] #8  0x00007fe20890c5ff in __curfile_search (cursor=0x7fe2116fa300) at src/third_party/wiredtiger/src/cursor/cur_file.c:200
 [2019/04/28 13:53:10.442] #9  0x00007fe208816751 in mongo::WiredTigerSizeStorer::load (this=0x7fe20cae87a0, uri=...) at src/mongo/db/storage/wiredtiger/wiredtiger_size_storer.cpp:108
 [2019/04/28 13:53:10.442] #10 0x00007fe208806935 in mongo::WiredTigerRecordStore::postConstructorInit (this=0x7fe249bc1600, opCtx=0x7fe21bd49280) at src/mongo/base/string_data.h:63
 [2019/04/28 13:53:10.442] #11 0x00007fe2087f384c in mongo::WiredTigerKVEngine::getGroupedRecordStore (this=0x7fe20cb46000, opCtx=0x7fe21bd49280, ns=..., ident="collection-9738-89034331986338385", options=..., prefix=...) at src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp:1226
 [2019/04/28 13:53:10.443] #12 0x00007fe208f13454 in mongo::KVCatalog::createCollection (this=0x7fe20cb3abe0, opCtx=opCtx@entry=0x7fe21bd49280, nss=..., options=..., allocateDefaultSpace=allocateDefaultSpace@entry=true) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/char_traits.h:350
 [2019/04/28 13:53:10.443] #13 0x00007fe208ed265b in mongo::DatabaseImpl::createCollection (this=<optimized out>, opCtx=0x7fe21bd49280, nss=..., options=..., createIdIndex=<optimized out>, idIndex=unowned empty BSONObj @ 0x7fe20a5c07ec <mongo::BSONObj::BSONObj()::kEmptyObjectPrototype>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/unique_ptr.h:342
 [2019/04/28 13:53:10.443] #14 0x00007fe208ed3718 in mongo::DatabaseImpl::userCreateNS (this=this@entry=0x7fe223819dc0, opCtx=0x7fe21bd49280, nss=..., collectionOptions=..., createDefaultIndexes=createDefaultIndexes@entry=true, idIndex=unowned empty BSONObj @ 0x7fe20a5c07ec <mongo::BSONObj::BSONObj()::kEmptyObjectPrototype>) at src/mongo/db/catalog/database_impl.cpp:930
 [2019/04/28 13:53:10.443] #15 0x00007fe208fa7077 in mongo::(anonymous namespace)::<lambda()>::operator()(void) const (__closure=0x7fe1d99bb8c0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/atomic_base.h:512
 [2019/04/28 13:53:10.443] #16 0x00007fe208fa80cd in mongo::writeConflictRetry<mongo::(anonymous namespace)::_createCollection(mongo::OperationContext*, const mongo::NamespaceString&, const mongo::CollectionOptions&, const mongo::BSONObj&)::<lambda()> > (f=..., ns=..., opStr=..., opCtx=0x7fe21bd49280) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/atomic_base.h:390
 [2019/04/28 13:53:10.443] #17 mongo::(anonymous namespace)::_createCollection (idIndex=unowned empty BSONObj @ 0x7fe20a5c07ec <mongo::BSONObj::BSONObj()::kEmptyObjectPrototype>, collectionOptions=..., nss=..., opCtx=<optimized out>) at src/mongo/db/catalog/create_collection.cpp:124
 [2019/04/28 13:53:10.443] #18 mongo::(anonymous namespace)::createCollection (opCtx=0x7fe21bd49280, nss=..., cmdObj=..., idIndex=unowned empty BSONObj @ 0x7fe20a5c07ec <mongo::BSONObj::BSONObj()::kEmptyObjectPrototype>, kind=<optimized out>) at src/mongo/db/catalog/create_collection.cpp:176
 [2019/04/28 13:53:10.443] #19 0x00007fe208fa84bc in mongo::createCollection (opCtx=opCtx@entry=0x7fe21bd49280, dbName="test13_fsmdb5", cmdObj=owned BSONObj 198 bytes @ 0x7fe257e8681d, idIndex=unowned empty BSONObj @ 0x7fe20a5c07ec <mongo::BSONObj::BSONObj()::kEmptyObjectPrototype>) at src/mongo/base/string_data.h:63
 [2019/04/28 13:53:10.443] #20 0x00007fe208e2bd47 in mongo::(anonymous namespace)::CmdCreate::run (this=<optimized out>, opCtx=0x7fe21bd49280, dbname="test13_fsmdb5", cmdObj=owned BSONObj 198 bytes @ 0x7fe257e8681d, result=...) at src/mongo/db/commands/dbcommands.cpp:394
 [2019/04/28 13:53:10.443] #21 0x00007fe209c92444 in mongo::BasicCommand::Invocation::run (this=0x7fe213832500, opCtx=0x7fe21bd49280, result=<optimized out>) at src/mongo/db/commands.cpp:592
 [2019/04/28 13:53:10.443] #22 0x00007fe208b60b65 in mongo::(anonymous namespace)::runCommandImpl (opCtx=<optimized out>, invocation=0x7fe213832500, request=..., replyBuilder=0x7fe227df5410, startOperationTime=..., behaviors=..., extraFieldsBuilder=0x7fe1d99bc3b0, sessionOptions=...) at src/mongo/db/service_entry_point_common.cpp:528
 [2019/04/28 13:53:10.443] #23 0x00007fe208b621cc in mongo::(anonymous namespace)::execCommandDatabase (opCtx=<optimized out>, command=0x7fe20b3e8dc0 <mongo::(anonymous namespace)::cmdCreate>, request=..., replyBuilder=<optimized out>, behaviors=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/unique_ptr.h:342
 [2019/04/28 13:53:10.443] #24 0x00007fe208b6302e in mongo::(anonymous namespace)::<lambda()>::operator()(void) const (__closure=0x7fe1d99bcbb0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/unique_ptr.h:342
 [2019/04/28 13:53:10.443] #25 0x00007fe208b638c0 in mongo::(anonymous namespace)::receivedCommands (behaviors=..., message=..., opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:901
 [2019/04/28 13:53:10.444] #26 mongo::ServiceEntryPointCommon::handleRequest (opCtx=0x7fe21bd49280, m=..., behaviors=...) at src/mongo/db/service_entry_point_common.cpp:1245
 [2019/04/28 13:53:10.444] #27 0x00007fe208b51f5c in mongo::ServiceEntryPointMongod::handleRequest (this=<optimized out>, opCtx=<optimized out>, m=...) at src/mongo/db/service_entry_point_common.h:59
 [2019/04/28 13:53:10.444] #28 0x00007fe208b5d82c in mongo::ServiceStateMachine::_processMessage (this=0x7fe255f78fd0, guard=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/unique_ptr.h:342
 [2019/04/28 13:53:10.444] #29 0x00007fe208b592ff in mongo::ServiceStateMachine::_runNextInGuard (this=0x7fe255f78fd0, guard=...) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.444] #30 0x00007fe208b5c5cd in mongo::ServiceStateMachine::<lambda()>::operator() (__closure=0x7fe21622bf40) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.444] #31 std::_Function_handler<void(), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:297
 [2019/04/28 13:53:10.444] #32 0x00007fe209995042 in std::function<void ()>::operator()() const (this=0x7fe1d99bd4c0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:682
 [2019/04/28 13:53:10.444] #33 mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName) (this=this@entry=0x7fe20c907a40, task=..., flags=flags@entry=mongo::transport::ServiceExecutor::kMayRecurse, taskName=taskName@entry=mongo::transport::ServiceExecutorTaskName::kSSMProcessMessage) at src/mongo/transport/service_executor_synchronous.cpp:107
 [2019/04/28 13:53:10.444] #34 0x00007fe208b5714d in mongo::ServiceStateMachine::_scheduleNextWithGuard (this=0x7fe255f78fd0, guard=..., flags=<optimized out>, taskName=<optimized out>, ownershipModel=mongo::ServiceStateMachine::Ownership::kOwned) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:87
 [2019/04/28 13:53:10.444] #35 0x00007fe208b5a0bd in mongo::ServiceStateMachine::_sourceCallback (this=0x7fe255f78fd0, status=Status::OK()) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.444] #36 0x00007fe208b585e2 in mongo::ServiceStateMachine::<lambda(mongo::StatusWith<mongo::Message>)>::operator() (__closure=<optimized out>, msg=StatusWith(OK, "")) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/atomic_base.h:512
 [2019/04/28 13:53:10.445] #37 mongo::future_details::call<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>&, mongo::Message> (arg=..., func=...) at src/mongo/util/future_impl.h:201
 [2019/04/28 13:53:10.445] #38 mongo::future_details::FutureImpl<mongo::Message>::<lambda(mongo::Message&&)>::operator() (this=<optimized out>, val=...) at src/mongo/util/future_impl.h:868
 [2019/04/28 13:53:10.445] #39 mongo::future_details::FutureImpl<mongo::Message>::generalImpl<mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Message&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Status&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda()> > (notReady=..., fail=..., success=..., this=0x7fe1d99bd800) at src/mongo/util/future_impl.h:1201
 [2019/04/28 13:53:10.445] #40 mongo::future_details::FutureImpl<mongo::Message>::getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> > (func=..., this=0x7fe1d99bd800) at src/mongo/util/future_impl.h:882
 [2019/04/28 13:53:10.445] #41 mongo::Future<mongo::Message>::getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> > (func=..., this=0x7fe1d99bd800) at src/mongo/util/future.h:348
 [2019/04/28 13:53:10.445] #42 mongo::ServiceStateMachine::_sourceMessage (this=0x7fe255f78fd0, guard=...) at src/mongo/transport/service_state_machine.cpp:319
 [2019/04/28 13:53:10.445] #43 0x00007fe208b5925b in mongo::ServiceStateMachine::_runNextInGuard (this=0x7fe255f78fd0, guard=...) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.445] #44 0x00007fe208b5c5cd in mongo::ServiceStateMachine::<lambda()>::operator() (__closure=0x7fe21c0a3540) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.445] #45 std::_Function_handler<void(), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:297
 [2019/04/28 13:53:10.445] #46 0x00007fe2099954ab in std::function<void ()>::operator()() const (this=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:682
 [2019/04/28 13:53:10.445] #47 mongo::transport::ServiceExecutorSynchronous::<lambda()>::operator() (__closure=0x7fe21dd78f50) at src/mongo/transport/service_executor_synchronous.cpp:124
 [2019/04/28 13:53:10.445] #48 std::_Function_handler<void(), mongo::transport::ServiceExecutorSynchronous::schedule(mongo::transport::ServiceExecutor::Task, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:297
 [2019/04/28 13:53:10.445] #49 0x00007fe20a065bd4 in std::function<void ()>::operator()() const (this=0x7fe219788800) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:682
 [2019/04/28 13:53:10.446] #50 mongo::(anonymous namespace)::runFunc (ctx=0x7fe219788800) at src/mongo/transport/service_entry_point_utils.cpp:56
 [2019/04/28 13:53:10.446] #51 0x00007fe2062200a4 in start_thread () from /lib64/libpthread.so.0
 [2019/04/28 13:53:10.446] #52 0x00007fe205f557fd in clone () from /lib64/libc.so.6

Eviction after commit:

 [2019/04/28 13:53:10.433] #0  0x00007fe206224408 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
 [2019/04/28 13:53:10.433] #1  0x00007fe20885635b in __wt_cond_wait_signal (session=session@entry=0x7fe20cb74450, cond=0x7fe2113a7080, usecs=usecs@entry=10000, run_func=run_func@entry=0x0, signalled=signalled@entry=0x7fe1d9dbf270) at src/third_party/wiredtiger/src/os_posix/os_mtx_cond.c:122
 [2019/04/28 13:53:10.433] #2  0x00007fe20884098c in __wt_cond_wait (run_func=0x0, usecs=10000, cond=<optimized out>, session=0x7fe20cb74450) at src/third_party/wiredtiger/src/include/misc.i:19
 [2019/04/28 13:53:10.433] #3  __wt_cache_eviction_worker (session=session@entry=0x7fe20cb74450, busy=<optimized out>, readonly=readonly@entry=false, pct_full=<optimized out>) at src/third_party/wiredtiger/src/evict/evict_lru.c:2424
 [2019/04/28 13:53:10.433] #4  0x00007fe2088997e9 in __wt_cache_eviction_check (busy=<optimized out>, readonly=false, didworkp=0x0, session=0x7fe20cb74450) at src/third_party/wiredtiger/src/include/cache.i:453
 [2019/04/28 13:53:10.433] #5  __wt_txn_commit (session=session@entry=0x7fe20cb74450, cfg=cfg@entry=0x7fe1d9dbf3c0) at src/third_party/wiredtiger/src/txn/txn.c:1042
 [2019/04/28 13:53:10.433] #6  0x00007fe20887b6b5 in __session_commit_transaction (wt_session=0x7fe20cb74450, config=<optimized out>) at src/third_party/wiredtiger/src/session/session_api.c:1664
 [2019/04/28 13:53:10.433] #7  0x00007fe2088101ad in mongo::WiredTigerRecoveryUnit::_txnClose (this=0x7fe2192428c0, commit=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/basic_string.h:2281
 [2019/04/28 13:53:10.433] #8  0x00007fe2088108cd in mongo::WiredTigerRecoveryUnit::_commit (this=0x7fe2192428c0) at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp:206
 [2019/04/28 13:53:10.433] #9  0x00007fe20881090a in mongo::WiredTigerRecoveryUnit::commitUnitOfWork (this=<optimized out>) at src/mongo/db/storage/wiredtiger/wiredtiger_recovery_unit.cpp:285
 [2019/04/28 13:53:10.433] #10 0x00007fe20a214234 in mongo::WriteUnitOfWork::commit (this=this@entry=0x7fe1d9dbf740) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/unique_ptr.h:342
 [2019/04/28 13:53:10.433] #11 0x00007fe208fd7fc1 in mongo::MultiIndexBlock::init(mongo::OperationContext*, mongo::Collection*, std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> > const&, std::function<mongo::Status (std::vector<mongo::BSONObj, std::allocator<mongo::BSONObj> >&)>) (this=this@entry=0x7fe1d9dbfd80, opCtx=<optimized out>, collection=<optimized out>, indexSpecs=std::vector of length 1, capacity 1 = {...}, onInit=...) at src/mongo/db/catalog/multi_index_block.cpp:346
 [2019/04/28 13:53:10.433] #12 0x00007fe208e20cda in mongo::(anonymous namespace)::<lambda()>::operator()(void) const (__closure=0x7fe1d9dbfbb0) at src/mongo/db/commands/create_indexes.cpp:378
 [2019/04/28 13:53:10.433] #13 0x00007fe208e235fd in mongo::writeConflictRetry<mongo::(anonymous namespace)::runCreateIndexes(mongo::OperationContext*, const string&, const mongo::BSONObj&, std::__cxx11::string&, mongo::BSONObjBuilder&, bool)::<lambda()> > (opStr=..., f=..., ns=..., opCtx=0x7fe222cb3b80) at src/mongo/db/operation_context.h:117
 [2019/04/28 13:53:10.433] #14 mongo::(anonymous namespace)::runCreateIndexes (opCtx=<optimized out>, dbname=..., cmdObj=..., errmsg=..., result=..., runTwoPhaseBuild=false) at src/mongo/db/commands/create_indexes.cpp:383
 [2019/04/28 13:53:10.433] #15 0x00007fe209c8dc66 in mongo::ErrmsgCommandDeprecated::run (this=<optimized out>, opCtx=<optimized out>, db=..., cmdObj=..., result=...) at src/mongo/db/commands.cpp:700
 [2019/04/28 13:53:10.433] #16 0x00007fe209c92444 in mongo::BasicCommand::Invocation::run (this=0x7fe21c0869c0, opCtx=0x7fe222cb3b80, result=<optimized out>) at src/mongo/db/commands.cpp:592
 [2019/04/28 13:53:10.433] #17 0x00007fe208b60b65 in mongo::(anonymous namespace)::runCommandImpl (opCtx=<optimized out>, invocation=0x7fe21c0869c0, request=..., replyBuilder=0x7fe213817e10, startOperationTime=..., behaviors=..., extraFieldsBuilder=0x7fe1d9dc03b0, sessionOptions=...) at src/mongo/db/service_entry_point_common.cpp:528
 [2019/04/28 13:53:10.433] #18 0x00007fe208b621cc in mongo::(anonymous namespace)::execCommandDatabase (opCtx=<optimized out>, command=0x7fe20b3e87a0 <mongo::(anonymous namespace)::cmdCreateIndex>, request=..., replyBuilder=<optimized out>, behaviors=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/unique_ptr.h:342
 [2019/04/28 13:53:10.433] #19 0x00007fe208b6302e in mongo::(anonymous namespace)::<lambda()>::operator()(void) const (__closure=0x7fe1d9dc0bb0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/unique_ptr.h:342
 [2019/04/28 13:53:10.433] #20 0x00007fe208b638c0 in mongo::(anonymous namespace)::receivedCommands (behaviors=..., message=..., opCtx=<optimized out>) at src/mongo/db/service_entry_point_common.cpp:901
 [2019/04/28 13:53:10.433] #21 mongo::ServiceEntryPointCommon::handleRequest (opCtx=0x7fe222cb3b80, m=..., behaviors=...) at src/mongo/db/service_entry_point_common.cpp:1245
 [2019/04/28 13:53:10.433] #22 0x00007fe208b51f5c in mongo::ServiceEntryPointMongod::handleRequest (this=<optimized out>, opCtx=<optimized out>, m=...) at src/mongo/db/service_entry_point_common.h:59
 [2019/04/28 13:53:10.433] #23 0x00007fe208b5d82c in mongo::ServiceStateMachine::_processMessage (this=0x7fe236534710, guard=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/unique_ptr.h:342
 [2019/04/28 13:53:10.433] #24 0x00007fe208b592ff in mongo::ServiceStateMachine::_runNextInGuard (this=0x7fe236534710, guard=...) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.433] #25 0x00007fe208b5c5cd in mongo::ServiceStateMachine::<lambda()>::operator() (__closure=0x7fe2247caee0) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.433] #26 std::_Function_handler<void(), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:297
 [2019/04/28 13:53:10.433] #27 0x00007fe209995042 in std::function<void ()>::operator()() const (this=0x7fe1d9dc14c0) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:682
 [2019/04/28 13:53:10.433] #28 mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName) (this=this@entry=0x7fe20c907a40, task=..., flags=flags@entry=mongo::transport::ServiceExecutor::kMayRecurse, taskName=taskName@entry=mongo::transport::ServiceExecutorTaskName::kSSMProcessMessage) at src/mongo/transport/service_executor_synchronous.cpp:107
 [2019/04/28 13:53:10.433] #29 0x00007fe208b5714d in mongo::ServiceStateMachine::_scheduleNextWithGuard (this=0x7fe236534710, guard=..., flags=<optimized out>, taskName=<optimized out>, ownershipModel=mongo::ServiceStateMachine::Ownership::kOwned) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:87
 [2019/04/28 13:53:10.433] #30 0x00007fe208b5a0bd in mongo::ServiceStateMachine::_sourceCallback (this=0x7fe236534710, status=Status::OK()) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.433] #31 0x00007fe208b585e2 in mongo::ServiceStateMachine::<lambda(mongo::StatusWith<mongo::Message>)>::operator() (__closure=<optimized out>, msg=StatusWith(OK, "")) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/atomic_base.h:512
 [2019/04/28 13:53:10.433] #32 mongo::future_details::call<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>&, mongo::Message> (arg=..., func=...) at src/mongo/util/future_impl.h:201
 [2019/04/28 13:53:10.433] #33 mongo::future_details::FutureImpl<mongo::Message>::<lambda(mongo::Message&&)>::operator() (this=<optimized out>, val=...) at src/mongo/util/future_impl.h:868
 [2019/04/28 13:53:10.433] #34 mongo::future_details::FutureImpl<mongo::Message>::generalImpl<mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Message&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda(mongo::Status&&)>, mongo::future_details::FutureImpl<T>::getAsync(Func&&) && [with Func = mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)>; T = mongo::Message]::<lambda()> > (notReady=..., fail=..., success=..., this=0x7fe1d9dc1800) at src/mongo/util/future_impl.h:1201
 [2019/04/28 13:53:10.433] #35 mongo::future_details::FutureImpl<mongo::Message>::getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> > (func=..., this=0x7fe1d9dc1800) at src/mongo/util/future_impl.h:882
 [2019/04/28 13:53:10.433] #36 mongo::Future<mongo::Message>::getAsync<mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)::<lambda(mongo::StatusWith<mongo::Message>)> > (func=..., this=0x7fe1d9dc1800) at src/mongo/util/future.h:348
 [2019/04/28 13:53:10.433] #37 mongo::ServiceStateMachine::_sourceMessage (this=0x7fe236534710, guard=...) at src/mongo/transport/service_state_machine.cpp:319
 [2019/04/28 13:53:10.433] #38 0x00007fe208b5925b in mongo::ServiceStateMachine::_runNextInGuard (this=0x7fe236534710, guard=...) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.433] #39 0x00007fe208b5c5cd in mongo::ServiceStateMachine::<lambda()>::operator() (__closure=0x7fe214850ce0) at src/mongo/transport/service_state_machine.cpp:198
 [2019/04/28 13:53:10.433] #40 std::_Function_handler<void(), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:297
 [2019/04/28 13:53:10.433] #41 0x00007fe2099954ab in std::function<void ()>::operator()() const (this=<optimized out>) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:682
 [2019/04/28 13:53:10.433] #42 mongo::transport::ServiceExecutorSynchronous::<lambda()>::operator() (__closure=0x7fe21de64090) at src/mongo/transport/service_executor_synchronous.cpp:124
 [2019/04/28 13:53:10.433] #43 std::_Function_handler<void(), mongo::transport::ServiceExecutorSynchronous::schedule(mongo::transport::ServiceExecutor::Task, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:297
 [2019/04/28 13:53:10.433] #44 0x00007fe20a065bd4 in std::function<void ()>::operator()() const (this=0x7fe215894b00) at /opt/mongodbtoolchain/revisions/94dac13bc8c0b50beff286acac77adeb2e81761e/stow/gcc-v3.6j8/include/c++/8.2.0/bits/std_function.h:682
 [2019/04/28 13:53:10.433] #45 mongo::(anonymous namespace)::runFunc (ctx=0x7fe215894b00) at src/mongo/transport/service_entry_point_utils.cpp:56
 [2019/04/28 13:53:10.433] #46 0x00007fe2062200a4 in start_thread () from /lib64/libpthread.so.0
 [2019/04/28 13:53:10.433] #47 0x00007fe205f557fd in clone () from /lib64/libc.so.6

Assignee:: Sulabh Mahajan

Reporter:: Sulabh Mahajan

Votes:: 0 Vote for this issue

Watchers:: 4 Start watching this issue

Created:: Jun 17 2019 05:35:34 AM UTC

Updated:: Oct 27 2023 08:20:51 PM UTC

Resolved:: Jun 19 2019 11:59:58 PM UTC

Details

Description

Attachments

Activity

People

Dates