[SERVER-31519] Server shutdown frequently Created: 11/Oct/17  Updated: 07/Nov/17  Resolved: 12/Oct/17

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 3.2.9
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Abhay Assignee: Mark Agarunov
Resolution: Done Votes: 0
Labels: MAREF
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive diagnostic-and-log.zip     PNG File ftdc.png    
Participants:

 Description   

Hello i have live website and have a separate DB server for mongodb under aws t2 medium instance. The server does not respond after some frequent time and to fix it i do a restart of the instance. Following is the log encountered in mongodb log before it gets shutdown

2017-10-10T07:20:54.722+0000 I NETWORK  [initandlisten] connection accepted from 52.221.108.123:49153 #23610 (3 connections now open)
2017-10-10T07:20:54.938+0000 F -        [conn23610] out of memory.
 
 0x13225f2 0x1322049 0x12a7e11 0xded676 0xcd2a79 0xcd8525 0x9baeac 0x12cedb5 0x7f6ba821f184 0x7f6ba7f4c37d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F225F2","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F22049","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"400000","o":"EA7E11","s":"_ZN5mongo11mongoMallocEm"},{"b":"400000","o":"9ED676","s":"_ZN5mongo7getMoreEPNS_16OperationContextEPKcixPbS4_"},{"b":"400000","o":"8D2A79","s":"_ZN5mongo15receivedGetMoreEPNS_16OperationContextERNS_10DbResponseERNS_7MessageERNS_5CurOpE"},{"b":"400000","o":"8D8525","s":"_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE"},{"b":"400000","o":"5BAEAC","s":"_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE"},{"b":"400000","o":"ECEDB5","s":"_ZN5mongo17PortMessageServer17handleIncomingMsgEPv"},{"b":"7F6BA8217000","o":"8184"},{"b":"7F6BA7E52000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.9", "gitVersion" : "22ec9e93b40c85fc7cae7d56e7d6a02fd811088c", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.13.0-100-generic", "version" : "#147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "78E57AF736DDF3E8C558F60DB63F68BCF686D70A" }, { "b" : "7FFD6D048000", "elfType" : 3, "buildId" : "5C010C1F1414642D6C56D676B45F7D4040811DE9" }, { "b" : "7F6BA9139000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "15A6A69B862FAC0AA589921DB1B0DF8A673FEC81" }, { "b" : "7F6BA8D5D000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "607C4A6F3428B6F065D55691AFDC4E343786A8A5" }, { "b" : "7F6BA8B55000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F6BA8951000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F6BA864B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F6BA8435000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F6BA8217000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F6BA7E52000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F6BA9398000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x13225f2]
 mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x79) [0x1322049]
 mongod(_ZN5mongo11mongoMallocEm+0x21) [0x12a7e11]
 mongod(_ZN5mongo7getMoreEPNS_16OperationContextEPKcixPbS4_+0x296) [0xded676]
 mongod(_ZN5mongo15receivedGetMoreEPNS_16OperationContextERNS_10DbResponseERNS_7MessageERNS_5CurOpE+0x1A9) [0xcd2a79]
 mongod(_ZN5mongo16assembleResponseEPNS_16OperationContextERNS_7MessageERNS_10DbResponseERKNS_11HostAndPortE+0xE35) [0xcd8525]
 mongod(_ZN5mongo16MyMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0xEC) [0x9baeac]
 mongod(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x325) [0x12cedb5]
 libpthread.so.0(+0x8184) [0x7f6ba821f184]
 libc.so.6(clone+0x6D) [0x7f6ba7f4c37d]
-----  END BACKTRACE  -----

Please help to overcome this issue



 Comments   
Comment by Abhay [ 13/Oct/17 ]

Okay mark then i think i will update the db version, thanks for your effort

Comment by Mark Agarunov [ 12/Oct/17 ]

Hello sharmasabhay,

Looking over the logs, I don't see a correlation between a specific query and the machine running out of memory.

Thanks,
Mark

Comment by Abhay [ 12/Oct/17 ]

Hello mark

Thanks for those links i will review them but did you find anything in the log file i have sent regarding the queries we are running i mean is it happening due to the queries we are firing in mongodb?

Comment by Mark Agarunov [ 12/Oct/17 ]

Hello sharmasabhay,

There are a few compatibility changes when upgrading from 3.2 to 3.4, however note that mongodb can be simply downgraded from 3.4 to 3.2, especially if the featureCompatibilityVersion has not been set to 3.4. Generally, we recommend a "burn in period" of running mongodb v3.4 before setting the featureCompatibilityVersion to 3.4:

It is recommended that after upgrading, you allow your deployment to run without enabling these features for a burn-in period to ensure the likelihood of downgrade is minimal. When you are confident that the likelihood of downgrade is minimal, enable these features.

Thanks,
Mark

Comment by Abhay [ 12/Oct/17 ]

Hello mark
Thanks for your effort

I believe swapping memory will not fix the issue but as your 2nd suggestion of updating version i think may work if you are sure about it . So will it be safe to update as the db server has live data i means is there any deprecated things which are in 3.2 but not in 3.4 just confirm me this so i can move accordingly.

Comment by Mark Agarunov [ 12/Oct/17 ]

Hello sharmasabhay,

To clarify my previous response, this behavior is likely due to the fact that the machine running mongod has only 1GB of memory. In Mongodb version 3.2, the minimum cache size, which correlates with memory usage, is 1GB, however this was lowered in version 3.4 to 256MB. Upgrading this node to mongodb version 3.4 should alleviate some of the memory pressure and reduce the likelihood of seeing this behavior.

Thanks,
Mark

Comment by Mark Agarunov [ 12/Oct/17 ]

Hello sharmasabhay,

Thank you for providing these files. Looking over the logs and diagnostic data, I'm not seeing anything that would indicate a bug in mongod. It appears that mongod is killed off by the kernel because the system is out of memory:

F -        [conn23610] out of memory.

According to the diagnostic data, mongod's memory usage is more or less stable:

As there is no substantial increase in memory usage, I suspect the system is simply running out of memory, possibly due to other processes running on the machine. I would suggest attempting to add a swap file to the system to alleviate some of the memory pressure and see if this remedies the issue.

Please note that SERVER project is for reporting bugs or feature suggestions for the MongoDB server. For MongoDB-related support discussion please post on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group.

See also our Technical Support page for additional support resources.

Thanks,
Mark

Comment by Abhay [ 11/Oct/17 ]

Hi mark

Thanks for the quick response, i have attached the mentioned files in a single zip please review and let me know asap.

Thanks
Abhay

Comment by Mark Agarunov [ 11/Oct/17 ]

Hello sharmasabhay,

Thank you for the report. Looking at the logs you've provided, this looks to be due to the server running out of memory, resulting in mongod being terminated. To get a better idea of why this may be happening, could you please provide the following:

  • The complete log files from mongod when this issue is present
  • Please archive (tar or zip) the $dbpath/diagnostic.data directory and attach it to this ticket

This should give some more insight into possible causes.

Thanks,
Mark

Generated at Thu Feb 08 04:27:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.