Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-58916

[connXXXX] out of memory crashing the server

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Incomplete
    • Affects Version/s: 4.2.6, 4.2.13
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Operating System:
      ALL
    • Steps To Reproduce:
      Hide

      So far, I was not able to reproduce the issue at will.

      I will see if I can get a core dump. FTDC data does not report anything unusual during the moment of the crash.

      Show
      So far, I was not able to reproduce the issue at will. I will see if I can get a core dump. FTDC data does not report anything unusual during the moment of the crash.

      Description

      Hi,

      I'm facing an issue where using MongoDB 4.2.12 (I also observed in 4.2.6) crashing with the following backtrace:

      mongod(mongo::printStackTrace(std::basic_ostream<char, std::char_traits<char> >&)+0x41) [0x55e8198e4f61]
      mongod(mongo::reportOutOfMemoryErrorAndExit()+0x87) [0x55e8198e4807]
      mongod(+0x2BC6631) [0x55e8199f0631]
      mongod(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0xC9) [0x55e819b79129]
      mongod(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)+0x61) [0x55e817c913d1]
      mongod(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_M_clone(std::allocator<char> const&, unsigned long)+0x1C) [0x55e817c9160c]
      mongod(mongo::rpc::getImpersonatedUserMetadata(mongo::OperationContext*)+0x22B) [0x55e81961a13b]
      mongod(mongo::CurOp::reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, bool, bool, mongo::BSONObjBuilder*)+0x3EA) [0x55e81923e47a]
      mongod(mongo::MongoInterfaceStandalone::_reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const+0x70) [0x55e8180dff80]
      mongod(mongo::MongoProcessCommon::getCurrentOps(boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::MongoProcessInterface::CurrentOpConnectionsMode, mongo::MongoProcessInterface::CurrentOpSessionsMode, mongo::MongoProcessInterface::CurrentOpUserMode, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpCursorMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const+0xFB) [0x55e8182f101b]
      mongod(mongo::DocumentSourceCurrentOp::getNext()+0x82A) [0x55e81917cefa]
      mongod(mongo::DocumentSourceMatch::getNext()+0x56) [0x55e8191ba1a6]
      mongod(mongo::DocumentSourceSingleDocumentTransformation::getNext()+0x38) [0x55e8191e7c98]
      mongod(mongo::Pipeline::getNext()+0x3D) [0x55e8191f983d]
      mongod(mongo::PipelineProxyStage::getNextBson()+0x35) [0x55e81875ab35]
      mongod(mongo::PipelineProxyStage::doWork(unsigned long*)+0x46) [0x55e81875a9a6]
      mongod(mongo::PlanStage::work(unsigned long*)+0x64) [0x55e81875b0e4]
      mongod(mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*, mongo::RecordId*)+0x22D) [0x55e8187a452d]
      mongod(mongo::PlanExecutorImpl::getNext(mongo::BSONObj*, mongo::RecordId*)+0x4D) [0x55e8187a4c8d]
      mongod(+0x16290E9) [0x55e8184530e9]
      mongod(mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, std::vector<mongo::Privilege, std::allocator<mongo::Privilege> > const&, mongo::rpc::ReplyBuilderInterface*)+0x2327) [0x55e8184574e7]
      mongod(+0x1620412) [0x55e81844a412]
      mongod(+0x12D5887) [0x55e8180ff887]
      mongod(+0x12D7BEB) [0x55e818101beb]
      mongod(mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)+0x4E7) [0x55e818102a07]
      mongod(mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)+0x3C) [0x55e8180efcdc]
      mongod(mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)+0xEF) [0x55e8180fc8bf]
      mongod(mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)+0xF6) [0x55e8180f9b66]
      mongod(+0x12D193B) [0x55e8180fb93b]
      mongod(mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)+0x182) [0x55e818fc9642]
      mongod(mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)+0x119) [0x55e8180f8159]
      mongod(mongo::ServiceStateMachine::_sourceCallback(mongo::Status)+0x4C4) [0x55e8180fa0a4]
      mongod(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)+0x2A6) [0x55e8180fac86]
      mongod(mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)+0xC2) [0x55e8180f9b32]
      mongod(+0x12D193B) [0x55e8180fb93b]
      mongod(+0x219FAA7) [0x55e818fc9aa7]
      mongod(+0x2826DD5) [0x55e819650dd5]
      mongod(+0x2826E34) [0x55e819650e34]
      libpthread.so.0(+0x7EA5) [0x7f02c0b96ea5]
      libc.so.6(clone+0x6D) [0x7f02c08bf8dd]
      

      Sometimes it also crashes with SIGSEGV.

      The query that was running:

      db.adminCommand( {"aggregate": 1, pipeline:[{"$currentOp": { "allUsers": true, "idleConnections": false }},
      {"$match": {"ns": {"$nin":["", "local.$cmd", "local.oplog.rs", "admin.$cmd.aggregate", "admin.$cmd", "local.replset.minvalid"]},
      "op": {"$nin":["","none"]}}},
      {"$addFields": {"currentOpTime":
      {"$dateFromString": {"dateString": "$currentOpTime"}}}}], "explain":true })
      

      Downgrading to 4.0 or upgrading solves the issue.

      Sometimes the connection and Mongo crashes with SIGSEV:

      2021-07-22T12:29:54.119+0900 F  -        [conn460933] Invalid access at address: 0
      2021-07-22T12:29:54.237+0900 F  -        [conn460933] Got signal: 11 (Segmentation fault).
      

        Attachments

          Activity

            People

            Assignee:
            dmitry.agranat Dmitry Agranat
            Reporter:
            vgrippa@gmail.com VINICIUS GRIPPA
            Participants:
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: