Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-58916

[connXXXX] out of memory crashing the server

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 4.2.6, 4.2.13
    • Component/s: None
    • Labels:
      None
    • ALL
    • Hide

      So far, I was not able to reproduce the issue at will.

      I will see if I can get a core dump. FTDC data does not report anything unusual during the moment of the crash.

      Show
      So far, I was not able to reproduce the issue at will. I will see if I can get a core dump. FTDC data does not report anything unusual during the moment of the crash.

      Hi,

      I'm facing an issue where using MongoDB 4.2.12 (I also observed in 4.2.6) crashing with the following backtrace:

      mongod(mongo::printStackTrace(std::basic_ostream<char, std::char_traits<char> >&)+0x41) [0x55e8198e4f61]
      mongod(mongo::reportOutOfMemoryErrorAndExit()+0x87) [0x55e8198e4807]
      mongod(+0x2BC6631) [0x55e8199f0631]
      mongod(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0xC9) [0x55e819b79129]
      mongod(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)+0x61) [0x55e817c913d1]
      mongod(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_M_clone(std::allocator<char> const&, unsigned long)+0x1C) [0x55e817c9160c]
      mongod(mongo::rpc::getImpersonatedUserMetadata(mongo::OperationContext*)+0x22B) [0x55e81961a13b]
      mongod(mongo::CurOp::reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, bool, bool, mongo::BSONObjBuilder*)+0x3EA) [0x55e81923e47a]
      mongod(mongo::MongoInterfaceStandalone::_reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const+0x70) [0x55e8180dff80]
      mongod(mongo::MongoProcessCommon::getCurrentOps(boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::MongoProcessInterface::CurrentOpConnectionsMode, mongo::MongoProcessInterface::CurrentOpSessionsMode, mongo::MongoProcessInterface::CurrentOpUserMode, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpCursorMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const+0xFB) [0x55e8182f101b]
      mongod(mongo::DocumentSourceCurrentOp::getNext()+0x82A) [0x55e81917cefa]
      mongod(mongo::DocumentSourceMatch::getNext()+0x56) [0x55e8191ba1a6]
      mongod(mongo::DocumentSourceSingleDocumentTransformation::getNext()+0x38) [0x55e8191e7c98]
      mongod(mongo::Pipeline::getNext()+0x3D) [0x55e8191f983d]
      mongod(mongo::PipelineProxyStage::getNextBson()+0x35) [0x55e81875ab35]
      mongod(mongo::PipelineProxyStage::doWork(unsigned long*)+0x46) [0x55e81875a9a6]
      mongod(mongo::PlanStage::work(unsigned long*)+0x64) [0x55e81875b0e4]
      mongod(mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*, mongo::RecordId*)+0x22D) [0x55e8187a452d]
      mongod(mongo::PlanExecutorImpl::getNext(mongo::BSONObj*, mongo::RecordId*)+0x4D) [0x55e8187a4c8d]
      mongod(+0x16290E9) [0x55e8184530e9]
      mongod(mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, std::vector<mongo::Privilege, std::allocator<mongo::Privilege> > const&, mongo::rpc::ReplyBuilderInterface*)+0x2327) [0x55e8184574e7]
      mongod(+0x1620412) [0x55e81844a412]
      mongod(+0x12D5887) [0x55e8180ff887]
      mongod(+0x12D7BEB) [0x55e818101beb]
      mongod(mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)+0x4E7) [0x55e818102a07]
      mongod(mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)+0x3C) [0x55e8180efcdc]
      mongod(mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)+0xEF) [0x55e8180fc8bf]
      mongod(mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)+0xF6) [0x55e8180f9b66]
      mongod(+0x12D193B) [0x55e8180fb93b]
      mongod(mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)+0x182) [0x55e818fc9642]
      mongod(mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)+0x119) [0x55e8180f8159]
      mongod(mongo::ServiceStateMachine::_sourceCallback(mongo::Status)+0x4C4) [0x55e8180fa0a4]
      mongod(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)+0x2A6) [0x55e8180fac86]
      mongod(mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)+0xC2) [0x55e8180f9b32]
      mongod(+0x12D193B) [0x55e8180fb93b]
      mongod(+0x219FAA7) [0x55e818fc9aa7]
      mongod(+0x2826DD5) [0x55e819650dd5]
      mongod(+0x2826E34) [0x55e819650e34]
      libpthread.so.0(+0x7EA5) [0x7f02c0b96ea5]
      libc.so.6(clone+0x6D) [0x7f02c08bf8dd]
      

      Sometimes it also crashes with SIGSEGV.

      The query that was running:

      db.adminCommand( {"aggregate": 1, pipeline:[{"$currentOp": { "allUsers": true, "idleConnections": false }},
      {"$match": {"ns": {"$nin":["", "local.$cmd", "local.oplog.rs", "admin.$cmd.aggregate", "admin.$cmd", "local.replset.minvalid"]},
      "op": {"$nin":["","none"]}}},
      {"$addFields": {"currentOpTime":
      {"$dateFromString": {"dateString": "$currentOpTime"}}}}], "explain":true })
      

      Downgrading to 4.0 or upgrading solves the issue.

      Sometimes the connection and Mongo crashes with SIGSEV:

      2021-07-22T12:29:54.119+0900 F  -        [conn460933] Invalid access at address: 0
      2021-07-22T12:29:54.237+0900 F  -        [conn460933] Got signal: 11 (Segmentation fault).
      

            Assignee:
            dmitry.agranat@mongodb.com Dmitry Agranat
            Reporter:
            vgrippa@gmail.com Vinicius Grippa
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: