Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-58916

[connXXXX] out of memory crashing the server

    XMLWordPrintableJSON

Details

    • Icon: Bug Bug
    • Resolution: Incomplete
    • Icon: Major - P3 Major - P3
    • None
    • 4.2.6, 4.2.13
    • None
    • None
    • ALL
    • Hide

      So far, I was not able to reproduce the issue at will.

      I will see if I can get a core dump. FTDC data does not report anything unusual during the moment of the crash.

      Show
      So far, I was not able to reproduce the issue at will. I will see if I can get a core dump. FTDC data does not report anything unusual during the moment of the crash.

    Description

      Hi,

      I'm facing an issue where using MongoDB 4.2.12 (I also observed in 4.2.6) crashing with the following backtrace:

      mongod(mongo::printStackTrace(std::basic_ostream<char, std::char_traits<char> >&)+0x41) [0x55e8198e4f61]
      mongod(mongo::reportOutOfMemoryErrorAndExit()+0x87) [0x55e8198e4807]
      mongod(+0x2BC6631) [0x55e8199f0631]
      mongod(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0xC9) [0x55e819b79129]
      mongod(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)+0x61) [0x55e817c913d1]
      mongod(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_M_clone(std::allocator<char> const&, unsigned long)+0x1C) [0x55e817c9160c]
      mongod(mongo::rpc::getImpersonatedUserMetadata(mongo::OperationContext*)+0x22B) [0x55e81961a13b]
      mongod(mongo::CurOp::reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, bool, bool, mongo::BSONObjBuilder*)+0x3EA) [0x55e81923e47a]
      mongod(mongo::MongoInterfaceStandalone::_reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const+0x70) [0x55e8180dff80]
      mongod(mongo::MongoProcessCommon::getCurrentOps(boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::MongoProcessInterface::CurrentOpConnectionsMode, mongo::MongoProcessInterface::CurrentOpSessionsMode, mongo::MongoProcessInterface::CurrentOpUserMode, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpCursorMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const+0xFB) [0x55e8182f101b]
      mongod(mongo::DocumentSourceCurrentOp::getNext()+0x82A) [0x55e81917cefa]
      mongod(mongo::DocumentSourceMatch::getNext()+0x56) [0x55e8191ba1a6]
      mongod(mongo::DocumentSourceSingleDocumentTransformation::getNext()+0x38) [0x55e8191e7c98]
      mongod(mongo::Pipeline::getNext()+0x3D) [0x55e8191f983d]
      mongod(mongo::PipelineProxyStage::getNextBson()+0x35) [0x55e81875ab35]
      mongod(mongo::PipelineProxyStage::doWork(unsigned long*)+0x46) [0x55e81875a9a6]
      mongod(mongo::PlanStage::work(unsigned long*)+0x64) [0x55e81875b0e4]
      mongod(mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*, mongo::RecordId*)+0x22D) [0x55e8187a452d]
      mongod(mongo::PlanExecutorImpl::getNext(mongo::BSONObj*, mongo::RecordId*)+0x4D) [0x55e8187a4c8d]
      mongod(+0x16290E9) [0x55e8184530e9]
      mongod(mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, std::vector<mongo::Privilege, std::allocator<mongo::Privilege> > const&, mongo::rpc::ReplyBuilderInterface*)+0x2327) [0x55e8184574e7]
      mongod(+0x1620412) [0x55e81844a412]
      mongod(+0x12D5887) [0x55e8180ff887]
      mongod(+0x12D7BEB) [0x55e818101beb]
      mongod(mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)+0x4E7) [0x55e818102a07]
      mongod(mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)+0x3C) [0x55e8180efcdc]
      mongod(mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)+0xEF) [0x55e8180fc8bf]
      mongod(mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)+0xF6) [0x55e8180f9b66]
      mongod(+0x12D193B) [0x55e8180fb93b]
      mongod(mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)+0x182) [0x55e818fc9642]
      mongod(mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)+0x119) [0x55e8180f8159]
      mongod(mongo::ServiceStateMachine::_sourceCallback(mongo::Status)+0x4C4) [0x55e8180fa0a4]
      mongod(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)+0x2A6) [0x55e8180fac86]
      mongod(mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)+0xC2) [0x55e8180f9b32]
      mongod(+0x12D193B) [0x55e8180fb93b]
      mongod(+0x219FAA7) [0x55e818fc9aa7]
      mongod(+0x2826DD5) [0x55e819650dd5]
      mongod(+0x2826E34) [0x55e819650e34]
      libpthread.so.0(+0x7EA5) [0x7f02c0b96ea5]
      libc.so.6(clone+0x6D) [0x7f02c08bf8dd]
      

      Sometimes it also crashes with SIGSEGV.

      The query that was running:

      db.adminCommand( {"aggregate": 1, pipeline:[{"$currentOp": { "allUsers": true, "idleConnections": false }},
      {"$match": {"ns": {"$nin":["", "local.$cmd", "local.oplog.rs", "admin.$cmd.aggregate", "admin.$cmd", "local.replset.minvalid"]},
      "op": {"$nin":["","none"]}}},
      {"$addFields": {"currentOpTime":
      {"$dateFromString": {"dateString": "$currentOpTime"}}}}], "explain":true })
      

      Downgrading to 4.0 or upgrading solves the issue.

      Sometimes the connection and Mongo crashes with SIGSEV:

      2021-07-22T12:29:54.119+0900 F  -        [conn460933] Invalid access at address: 0
      2021-07-22T12:29:54.237+0900 F  -        [conn460933] Got signal: 11 (Segmentation fault).
      

      Attachments

        Activity

          People

            dmitry.agranat@mongodb.com Dmitry Agranat
            vgrippa@gmail.com Vinicius Grippa
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: