[SERVER-34712] mongodb out of memory error Created: 27/Apr/18  Updated: 04/Jun/18  Resolved: 04/Jun/18

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Guru chandra boopathi Assignee: Dmitry Agranat
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File backtrace.txt     Text File backtrace.txt     Text File backtrace.txt     Text File backtrace.txt     File diagnostic.data.7z     File diagnostic.data.tar.gz     File diagnostic.data.tar.gz    
Participants:

 Description   

MongoDB went down because out of memory. But I have enough ram space and disk space. I received the following error in log file.



 Comments   
Comment by Dmitry Agranat [ 04/Jun/18 ]

Hi guruchandra,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Dima

Comment by Dmitry Agranat [ 27/May/18 ]

Hi guruchandra,

We still need the information I've requested in my last comment to diagnose the problem. If this is still an issue for you, can you please upload this data after enabling the heap profiler:

  • complete log files and diagnostic.data from the affected mongod
  • the output of the top command

Thanks,
Dima

Comment by Dmitry Agranat [ 03/May/18 ]

Hi guruchandra

I've reviewed the data you've provided. mongod appears to be increasing its memory consumption outside of the WiredTiger cache. We've been able to rule out a number of common explanations for this behavior such as an increase in connections or a large number of open cursors, but do not have enough information to conclusively determine the root cause.

To help us continue to investigate, would you be willing to restart an affected mongod with the heap profiler enabled and run it for a week or until the next occurrence of OOM?

mongod --setParameter heapProfilingEnabled=true

This will allow us to identify which threads are accumulating memory. Please note that this is a diagnostic setting and we have observed a 10-30% performance impact on typical CPU-bound workloads. After reviewing the diagnostic.data, I believe that enabling this setting should not significantly harm your performance, and will provide the data to help us determine the root cause. However, to be safe, I would recommend more carefully monitoring the mongod after enabling the heap profiler.

I've created a secure upload portal for you to use. After collecting the additional diagnostic.data please upload the following:

  • complete log files of the affected mongod
  • diagnostic.data

Lastly, could you clarify what other processes are running on this server? You can collect this information by executing top for a few minutes:

delay=1  # collection interval, in seconds
top -b -d ${delay:?} >> /tmp/top_$(hostname)_$(date +%Y%m%d%H%M).log

Thanks,
Dima

Comment by Guru chandra boopathi [ 02/May/18 ]

Hi Dima,

sorry , i was uploaded the wrong one.pls find the attachment now 

Comment by Guru chandra boopathi [ 02/May/18 ]

diagnostic.data.7z backtrace.txt

Comment by Dmitry Agranat [ 01/May/18 ]

Hi guruchandra

The attached diagnostics.data is empty except for the awslogs-agent-setup.py file.

Please archive the diagnostic.data directory under the node's $dbpath and upload it to this ticket.

Thanks,
Dima

Comment by Guru chandra boopathi [ 30/Apr/18 ]

hi Kelsey,

Please find the attached backtrace and diagnostic.data . i don't have syslog for particular time.

Comment by Kelsey Schubert [ 27/Apr/18 ]

Hi guruchandra,

Thank you for reporting this issue. So we can continue to investigate, would you please provide the the following information?

  • complete mongod logs covering at least a few hours before the the event until the mongod is shutdown (e.g. the full backtrace)
  • syslogs around the time of incident
  • an archive of the diagnostic.data directory in the affected node's $dbpath

Thanks again,
Kelsey

Comment by Guru chandra boopathi [ 27/Apr/18 ]

Error message

2018-04-26T16:29:48.205+0000 F - [repl writer worker 2] out of memory.
 
0x559faf86d651 0x559faf86cc84 0x559fafa29f2b 0x559fb036470c 0x559faf469494 
0x559faee57883 0x559faecf56f9 0x559faf2792ff 0x559faf2796bf 0x559faf27b897 
0x559faf35e020 0x 559faf357291 0x559faf357bdd 0x559faf357ced 0x559faf35c7d8 
0x559faf35dc13 0x559faf35329f 0x559faf35438a 0x559faf7e755c 0x559faf7e800c 
0x559faf7e89f6 0x559fb02e2b50 0x7fc 00e7c56ba 0x7fc00e4fb41d 
----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"559FAE2FC000","o":"1571651","s":"_ZN5mongo15printStackTraceERSo"},
{"b":"559FAE2FC000","o":"1570C84","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv" },
{"b":"559FAE2FC000","o":"172DF2B"},
{"b":"559FAE2FC000","o":"206870C","s":"tc_new"},
{"b":"559FAE2FC000","o":"116D494","s":"_ZN5mongo3Top6recordEPNS_16OperationContextE NS_10StringDataENS_9LogicalOpEixbNS_7Command13ReadWriteTypeE"},
{"b":"559FAE2FC000","o":"B5B883","s":"_ZN5mongo16OldClientContextD1Ev"},
{"b":"559FAE2FC000","o":"9F96F9", "s":"_ZN5mongo16createCollectionEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjESC_"},
{"b":"559FAE2FC000","o":"F7D2FF"},
{"b" :"559FAE2FC000","o":"F7D6BF"},
{"b":"559FAE2FC000","o":"F7F897","s":"_ZN5mongo4repl19applyCommand_inlockEPNS_16OperationContextERKNS_7BSONObjEb"},
{"b":"559FAE2FC000","o" :"1062020","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextERKNS0_7BSONObjEbEPS7_E9_M_invokeERKSt9_Any_dataOS3_S6_Ob"},
{"b":"559FAE2FC000","o":"105 B291","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bS7_IFvvEEEES7_IFS8_S3_S6_bEESC_"

Generated at Thu Feb 08 04:37:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.