Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-16794

Invariant failure: WT_NOTFOUND error with concurrent removes and updates

    • Type: Icon: Bug Bug
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.8.0-rc5
    • Affects Version/s: 2.8.0-rc3, 2.8.0-rc4
    • Component/s: Storage
    • Labels:
    • Fully Compatible
    • ALL
    • 0

      This invariant failure is triggered by concurrent removes and updates. (I was able to trigger a similar invariant failure with rc3, so it seems unrelated to yielding.)

      SERVER-16121 might be a similar issue.

      Invariant failure: ret resulted in status UnknownError -31803: WT_NOTFOUND: item not found at src/mongo/db/storage/wiredtiger/wiredtiger_record_store.cpp 553
      
      frame #0: 0x00007fff85ae0976 libsystem_kernel.dylib`__write_nocancel + 10
      frame #1: 0x00007fff90b39a0d libsystem_c.dylib`_swrite + 87
      frame #2: 0x00007fff90b324c7 libsystem_c.dylib`__sflush + 101
      frame #3: 0x00007fff90b34c71 libsystem_c.dylib`__sfvwrite + 454
      frame #4: 0x00007fff90b350cb libsystem_c.dylib`fwrite + 142
      frame #5: 0x00007fff87130954 libstdc++.6.dylib`std::ostream::_M_write(char const*, long) + 34
      frame #6: 0x00007fff871309b0 libstdc++.6.dylib`std::ostream::write(char const*, long) + 54
      frame #7: 0x000000010002dc71 mongod`mongo::operator<<(stream=0x00007fff72d1bc30, value=0x00000001095b5688) + 65 at string_data.cpp:57
      frame #8: 0x000000010088d0b2 mongod`mongo::logger::MessageEventDetailsEncoder::encode(this=0x0000000103d17fc0, event=0x00000001095b5950, os=0x00007fff72d1bc30) + 690 at message_event_utf8_encoder.cpp:79
      frame #9: 0x0000000100447a46 mongod`mongo::logger::ConsoleAppender<mongo::logger::MessageEventEphemeral, mongo::Console>::append(this=0x0000000103d17eb0, event=0x00000001095b5950) + 134 at console_appender.h:54
      frame #10: 0x000000010088d88b mongod`mongo::logger::LogDomain<mongo::logger::MessageEventEphemeral>::append(this=0x0000000103d17f20, event=0x00000001095b5950) + 171 at log_domain-impl.h:60
      frame #11: 0x000000010088a47a mongod`mongo::logger::LogstreamBuilder::~LogstreamBuilder(this=0x00000001095b5a40) + 410 at logstream_builder.cpp:123
      frame #12: 0x000000010088a8e5 mongod`mongo::logger::LogstreamBuilder::~LogstreamBuilder(this=0x00000001095b5a40) + 21 at logstream_builder.cpp:117
      frame #13: 0x0000000100a75989 mongod`mongo::invariantOKFailed(expr=0x00000001015efeab, status=0x00000001095b5bf8, file=0x00000001015f086b, line=553) + 425 at assert_util.cpp:158
      frame #14: 0x0000000100868416 mongod`mongo::WiredTigerRecordStore::updateRecord(this=0x0000000103b62770, txn=0x00000001095bb910, loc=0x00000001095b6540, data=0x000000010683cfa4, len=58, enforceQuota=false, notifier=0x0000000103b61338) + 470 at wiredtiger_record_store.cpp:553
      frame #15: 0x00000001001944b5 mongod`mongo::Collection::updateDocument(this=0x0000000103b61330, txn=0x00000001095bb910, oldLocation=0x00000001095b6540, objOld=0x00000001095b6530, objNew=0x00000001095b6248, enforceQuota=true, indexesAffected=false, debug=0x00000001095b71b0) + 1621 at collection.cpp:351
      frame #16: 0x0000000100397461 mongod`mongo::UpdateStage::transformAndUpdate(this=0x0000000103d5feb0, oldObj=0x00000001095b6530, loc=0x00000001095b6540) + 3073 at update.cpp:564
      frame #17: 0x000000010039ab66 mongod`mongo::UpdateStage::work(this=0x0000000103d5feb0, out=0x00000001095b66d0) + 1910 at update.cpp:819
      frame #18: 0x00000001005fe470 mongod`mongo::PlanExecutor::getNext(this=0x0000000103d39430, objOut=0x00000001095b6798, dlOut=0x0000000000000000) + 464 at plan_executor.cpp:305
      frame #19: 0x00000001005fed74 mongod`mongo::PlanExecutor::executePlan(this=0x0000000103d39430) + 84 at plan_executor.cpp:417
      frame #20: 0x000000010029bf8a mongod`mongo::multiUpdate(txn=0x00000001095bb910, updateItem=0x00000001095b8410, result=0x00000001095b6ef0)::WriteOpResult*) + 3418 at batch_executor.cpp:1294
      frame #21: 0x0000000100299eb4 mongod`mongo::WriteBatchExecutor::execUpdate(this=0x00000001095b8948, updateItem=0x00000001095b8410, upsertedId=0x00000001095b8420, error=0x00000001095b8430) + 276 at batch_executor.cpp:899
      frame #22: 0x0000000100298f61 mongod`mongo::WriteBatchExecutor::bulkExecute(this=0x00000001095b8948, request=0x00000001095b89e8, upsertedIds=0x00000001095b8798, errors=0x00000001095b87b8) + 209 at batch_executor.cpp:767
      frame #23: 0x000000010029811e mongod`mongo::WriteBatchExecutor::executeBatch(this=0x00000001095b8948, request=0x00000001095b89e8, response=0x00000001095b8a58) + 1774 at batch_executor.cpp:268
      frame #24: 0x00000001002a6c55 mongod`mongo::WriteCmd::run(this=0x0000000103b57460, txn=0x00000001095bb910, dbName=0x00000001095b9478, cmdObj=0x00000001095b9848, options=0, errMsg=0x00000001095b90d0, result=0x00000001095ba890, fromRepl=false) + 725 at write_commands.cpp:144
      frame #25: 0x00000001002cd475 mongod`mongo::_execCommand(txn=0x00000001095bb910, c=0x0000000103b57460, dbname=0x00000001095b9478, cmdObj=0x00000001095b9848, queryOptions=0, errmsg=0x00000001095b90d0, result=0x00000001095ba890, fromRepl=false) + 149 at dbcommands.cpp:1252
      frame #26: 0x00000001002cebaf mongod`mongo::Command::execCommand(txn=0x00000001095bb910, c=0x0000000103b57460, queryOptions=0, cmdns=0x0000000105025014, cmdObj=0x00000001095b9848, result=0x00000001095ba890, fromRepl=false) + 4767 at dbcommands.cpp:1468
      frame #27: 0x00000001002cf842 mongod`mongo::_runCommands(txn=0x00000001095bb910, ns=0x0000000105025014, _cmdobj=0x00000001095baf20, b=0x00000001095ba8e8, anObjBuilder=0x00000001095ba890, fromRepl=false, queryOptions=0) + 2066 at dbcommands.cpp:1543
      frame #28: 0x00000001005b2810 mongod`mongo::runCommands(txn=0x00000001095bb910, ns=0x0000000105025014, jsobj=0x00000001095baf20, curop=0x0000000106008800, b=0x00000001095ba8e8, anObjBuilder=0x00000001095ba890, fromRepl=false, queryOptions=0) + 96 at find.cpp:135
      frame #29: 0x00000001005af059 mongod`mongo::runQuery(txn=0x00000001095bb910, m=0x00000001095bbcf0, q=0x00000001095baf08, curop=0x0000000106008800, result=0x0000000103d34d50, fromDBDirectClient=false) + 1433 at find.cpp:569
      frame #30: 0x000000010044b653 mongod`mongo::receivedQuery(txn=0x00000001095bb910, c=0x0000000106808c70, dbresponse=0x00000001095bb8b8, m=0x00000001095bbcf0, fromDBDirectClient=false) + 915 at instance.cpp:212
      frame #31: 0x00000001004496af mongod`mongo::assembleResponse(txn=0x00000001095bb910, m=0x00000001095bbcf0, dbresponse=0x00000001095bb8b8, remote=0x00000001095bb8a8, fromDBDirectClient=false) + 1327 at instance.cpp:395
      frame #32: 0x0000000100013b5f mongod`mongo::MyMessageHandler::process(this=0x0000000103f125a0, m=0x00000001095bbcf0, port=0x0000000103f50b90, le=0x000000010680a720) + 351 at db.cpp:197
      frame #33: 0x0000000100aad65c mongod`mongo::PortMessageServer::handleIncomingMsg(arg=0x0000000103f50b90) + 2828 at message_server_port.cpp:225
      frame #34: 0x0000000100aabad9 mongod`void* boost::_bi::list1<boost::_bi::value<mongo::(anonymous namespace)::MessagingPortWithHandler*> >::operator(this=0x0000000103f51040, (null)=type<void *> at 0x00000001095bbdf8, f=0x0000000103f51038, a=0x00000001095bbe20, (null)=0)<void*, void* (*)(void*), boost::_bi::list0>(boost::_bi::type<void*>, void* (*&)(void*), boost::_bi::list0&, long) + 89 at bind.hpp:243
      frame #35: 0x0000000100aaba74 mongod`boost::_bi::bind_t<void*, void* (this=0x0000000103f51038)(void*), boost::_bi::list1<boost::_bi::value<mongo::(anonymous namespace)::MessagingPortWithHandler*> > >::operator()() + 68 at bind_template.hpp:20
      frame #36: 0x0000000100aaba1c mongod`boost::detail::thread_data<boost::_bi::bind_t<void*, void* (this=0x0000000103f50e50)(void*), boost::_bi::list1<boost::_bi::value<mongo::(anonymous namespace)::MessagingPortWithHandler*> > > >::run() + 28 at thread.hpp:62
      frame #37: 0x0000000100b4c644 mongod`boost::(anonymous namespace)::thread_proxy(param=0x0000000103f50e50) + 148 at thread.cpp:121
      frame #38: 0x00007fff90b862fc libsystem_pthread.dylib`_pthread_body + 131
      frame #39: 0x00007fff90b86279 libsystem_pthread.dylib`_pthread_start + 176
      frame #40: 0x00007fff90b844b1 libsystem_pthread.dylib`thread_start + 13
      

      Version: bc274534b82eb2328d417dc5b256fb6d66287f74

            Assignee:
            geert.bosch@mongodb.com Geert Bosch
            Reporter:
            kamran.khan Kamran K.
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: