[SERVER-58916] [connXXXX] out of memory crashing the server Created: 28/Jul/21  Updated: 24/Aug/21  Resolved: 24/Aug/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 4.2.6, 4.2.13
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Vinicius Grippa Assignee: Dmitry Agranat
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Operating System: ALL
Steps To Reproduce:

So far, I was not able to reproduce the issue at will.

I will see if I can get a core dump. FTDC data does not report anything unusual during the moment of the crash.

Participants:

 Description   

Hi,

I'm facing an issue where using MongoDB 4.2.12 (I also observed in 4.2.6) crashing with the following backtrace:

mongod(mongo::printStackTrace(std::basic_ostream<char, std::char_traits<char> >&)+0x41) [0x55e8198e4f61]
mongod(mongo::reportOutOfMemoryErrorAndExit()+0x87) [0x55e8198e4807]
mongod(+0x2BC6631) [0x55e8199f0631]
mongod(tcmalloc::allocate_full_cpp_throw_oom(unsigned long)+0xC9) [0x55e819b79129]
mongod(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&)+0x61) [0x55e817c913d1]
mongod(std::basic_string<char, std::char_traits<char>, std::allocator<char> >::_Rep::_M_clone(std::allocator<char> const&, unsigned long)+0x1C) [0x55e817c9160c]
mongod(mongo::rpc::getImpersonatedUserMetadata(mongo::OperationContext*)+0x22B) [0x55e81961a13b]
mongod(mongo::CurOp::reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, bool, bool, mongo::BSONObjBuilder*)+0x3EA) [0x55e81923e47a]
mongod(mongo::MongoInterfaceStandalone::_reportCurrentOpForClient(mongo::OperationContext*, mongo::Client*, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const+0x70) [0x55e8180dff80]
mongod(mongo::MongoProcessCommon::getCurrentOps(boost::intrusive_ptr<mongo::ExpressionContext> const&, mongo::MongoProcessInterface::CurrentOpConnectionsMode, mongo::MongoProcessInterface::CurrentOpSessionsMode, mongo::MongoProcessInterface::CurrentOpUserMode, mongo::MongoProcessInterface::CurrentOpTruncateMode, mongo::MongoProcessInterface::CurrentOpCursorMode, mongo::MongoProcessInterface::CurrentOpBacktraceMode) const+0xFB) [0x55e8182f101b]
mongod(mongo::DocumentSourceCurrentOp::getNext()+0x82A) [0x55e81917cefa]
mongod(mongo::DocumentSourceMatch::getNext()+0x56) [0x55e8191ba1a6]
mongod(mongo::DocumentSourceSingleDocumentTransformation::getNext()+0x38) [0x55e8191e7c98]
mongod(mongo::Pipeline::getNext()+0x3D) [0x55e8191f983d]
mongod(mongo::PipelineProxyStage::getNextBson()+0x35) [0x55e81875ab35]
mongod(mongo::PipelineProxyStage::doWork(unsigned long*)+0x46) [0x55e81875a9a6]
mongod(mongo::PlanStage::work(unsigned long*)+0x64) [0x55e81875b0e4]
mongod(mongo::PlanExecutorImpl::_getNextImpl(mongo::Snapshotted<mongo::BSONObj>*, mongo::RecordId*)+0x22D) [0x55e8187a452d]
mongod(mongo::PlanExecutorImpl::getNext(mongo::BSONObj*, mongo::RecordId*)+0x4D) [0x55e8187a4c8d]
mongod(+0x16290E9) [0x55e8184530e9]
mongod(mongo::runAggregate(mongo::OperationContext*, mongo::NamespaceString const&, mongo::AggregationRequest const&, mongo::BSONObj const&, std::vector<mongo::Privilege, std::allocator<mongo::Privilege> > const&, mongo::rpc::ReplyBuilderInterface*)+0x2327) [0x55e8184574e7]
mongod(+0x1620412) [0x55e81844a412]
mongod(+0x12D5887) [0x55e8180ff887]
mongod(+0x12D7BEB) [0x55e818101beb]
mongod(mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, mongo::ServiceEntryPointCommon::Hooks const&)+0x4E7) [0x55e818102a07]
mongod(mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)+0x3C) [0x55e8180efcdc]
mongod(mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)+0xEF) [0x55e8180fc8bf]
mongod(mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)+0xF6) [0x55e8180f9b66]
mongod(+0x12D193B) [0x55e8180fb93b]
mongod(mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)+0x182) [0x55e818fc9642]
mongod(mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)+0x119) [0x55e8180f8159]
mongod(mongo::ServiceStateMachine::_sourceCallback(mongo::Status)+0x4C4) [0x55e8180fa0a4]
mongod(mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)+0x2A6) [0x55e8180fac86]
mongod(mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)+0xC2) [0x55e8180f9b32]
mongod(+0x12D193B) [0x55e8180fb93b]
mongod(+0x219FAA7) [0x55e818fc9aa7]
mongod(+0x2826DD5) [0x55e819650dd5]
mongod(+0x2826E34) [0x55e819650e34]
libpthread.so.0(+0x7EA5) [0x7f02c0b96ea5]
libc.so.6(clone+0x6D) [0x7f02c08bf8dd]

Sometimes it also crashes with SIGSEGV.

The query that was running:

db.adminCommand( {"aggregate": 1, pipeline:[{"$currentOp": { "allUsers": true, "idleConnections": false }},
{"$match": {"ns": {"$nin":["", "local.$cmd", "local.oplog.rs", "admin.$cmd.aggregate", "admin.$cmd", "local.replset.minvalid"]},
"op": {"$nin":["","none"]}}},
{"$addFields": {"currentOpTime":
{"$dateFromString": {"dateString": "$currentOpTime"}}}}], "explain":true })

Downgrading to 4.0 or upgrading solves the issue.

Sometimes the connection and Mongo crashes with SIGSEV:

2021-07-22T12:29:54.119+0900 F  -        [conn460933] Invalid access at address: 0
2021-07-22T12:29:54.237+0900 F  -        [conn460933] Got signal: 11 (Segmentation fault).



 Comments   
Comment by Dmitry Agranat [ 24/Aug/21 ]

Hi vgrippa@gmail.com,

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Vinicius Grippa [ 11/Aug/21 ]

Hi Dimitry,

Sorry, I do not have the FTDC data. If the error appears again I will submit it.

Thanks for your attention.

Comment by Dmitry Agranat [ 11/Aug/21 ]

Hi vgrippa@gmail.com,

We still need additional information to diagnose the problem. If this is still an issue for you, would you please upload it into this support uploader?

Regards,
Dima

Comment by Dmitry Agranat [ 29/Jul/21 ]

Hi vgrippa@gmail.com,

Would you please archive (tar or zip) the mongod.log files covering the incident and the $dbpath/diagnostic.data directory (the contents are described here) and upload them to this support uploader location?

Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time.

Thanks,
Dima

Generated at Thu Feb 08 05:45:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.