[SERVER-37439] Out of memory exception Created: 03/Oct/18  Updated: 25/Oct/18  Resolved: 25/Oct/18

Status: Closed
Project: Core Server
Component/s: Stability
Affects Version/s: 3.6.7
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Travis Brown Assignee: Kelsey Schubert
Resolution: Incomplete Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2018-10-03 at 10.18.07.png    
Operating System: ALL
Steps To Reproduce:

Run mongodb in a 3 server cluster for ~30 days

Participants:

 Description   

mongodb 3.6.7 will use nearly all the memory in the system when running as primary, and eventually crash with an out of memory exception. This machine has 32GB of RAM and is running Ubuntu 16.04.5 LTS.

 

2018-10-03T13:38:15.588+0000 I NETWORK  [conn1553489] end connection 172.20.0.1:59480 (113 connections now open)
2018-10-03T13:38:15.689+0000 I NETWORK  [conn1553490] end connection 10.150.1.1:55372 (113 connections now open)
2018-10-03T13:38:17.184+0000 I NETWORK  [conn1553491] received client metadata from 172.20.0.2:35972 conn1553491: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.6" }, os: { type: "Linux", name: "Ubu
ntu", architecture: "x86_64", version: "14.04" } }
2018-10-03T13:38:17.566+0000 I NETWORK  [conn1553492] received client metadata from 172.20.0.22:54262 conn1553492: { driver: { name: "PyMongo", version: "3.5.1" }, os: { type: "Linux", name: "Ubuntu 16.04 xenial", archite
cture: "x86_64", version: "4.17.17-x86_64-linode116" }, platform: "CPython 2.7.15.final.0" }
2018-10-03T13:38:17.832+0000 F -        [conn290616] out of memory.
 
 0x55bf275693b1 0x55bf275689e4 0x55bf2755e341 0x55bf25d2de7e 0x55bf2701d730 0x55bf25f53e38 0x55bf25f5549b 0x55bf25f562a1 0x55bf25f627ea 0x55bf25f5e1a7 0x55bf25f615e1 0x55bf26e64022 0x55bf25f5d00f 0x55bf25f5f555 0x55bf25f5
fe4b 0x55bf25f5e22d 0x55bf25f615e1 0x55bf26e64585 0x55bf27422584 0x7f93edae76ba 0x7f93ed81d41d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55BF25330000","o":"22393B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55BF25330000","o":"22389E4","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"55BF25330000","o":"222E341","s":"_ZN5mongo12m
ongoReallocEPvm"},{"b":"55BF25330000","o":"9FDE7E","s":"_ZN5mongo11_BufBuilderINS_21SharedBufferAllocatorEE15grow_reallocateEi"},{"b":"55BF25330000","o":"1CED730","s":"_ZN5mongo3rpc17OpMsgReplyBuilder22getInPlaceReplyBuil
derEm"},{"b":"55BF25330000","o":"C23E38"},{"b":"55BF25330000","o":"C2549B"},{"b":"55BF25330000","o":"C262A1","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55BF2533000
0","o":"C327EA","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55BF25330000","o":"C2E1A7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55BF25330000","o":
"C315E1"},{"b":"55BF25330000","o":"1B34022","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55BF25330000","o":"C2D
00F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55BF25330000","o":"C2F555","s":"_ZN5mo
ngo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55BF25330000","o":"C2FE4B","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55BF25330000","o":"C2E22D","s":"_ZN5mongo19ServiceSta
teMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55BF25330000","o":"C315E1"},{"b":"55BF25330000","o":"1B34585"},{"b":"55BF25330000","o":"20F2584"},{"b":"7F93EDAE0000","o":"76BA"},{"b":"7F93ED716000","o":"10741D","s":
"clone"}],"processInfo":{ "mongodbVersion" : "3.6.7", "gitVersion" : "2628472127e9f1826e02c665c1d93880a204075e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.17.17-x86_64-linode116", "version" :
 "#1 SMP PREEMPT Mon Aug 20 16:07:40 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55BF25330000", "elfType" : 3, "buildId" : "5D78F445F57AA961C35C97316819BF42C1939FFF" }, { "b" : "7FFFA2BB7000", "elfType" : 3, "
buildId" : "20B50035DDDF6D83A1FA4A5C699462D404FFB672" }, { "b" : "7F93EECD5000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }, { "b" : "7F93EE891
000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "250E875F74377DFC74DE48BF80CCB237BB4EFF1D" }, { "b" : "7F93EE628000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3,
"buildId" : "513282AC7EB386E2C0133FD9E1B6B8A0F38B047D" }, { "b" : "7F93EE424000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7F93EE21C000
", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7F93EDF13000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB8
5DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7F93EDCFD000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F93EDAE0000", "path" : "/
lib/x86_64-linux-gnu/libpthread.so.0", "elfTy
pe" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7F93ED716000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7F93EEEF0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55bf275693b1]
 mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x84) [0x55bf275689e4]
 mongod(_ZN5mongo12mongoReallocEPvm+0x21) [0x55bf2755e341]
 mongod(_ZN5mongo11_BufBuilderINS_21SharedBufferAllocatorEE15grow_reallocateEi+0x5E) [0x55bf25d2de7e]
 mongod(_ZN5mongo3rpc17OpMsgReplyBuilder22getInPlaceReplyBuilderEm+0x80) [0x55bf2701d730]
 mongod(+0xC23E38) [0x55bf25f53e38]
 mongod(+0xC2549B) [0x55bf25f5549b]
 mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2A1) [0x55bf25f562a1]
 mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55bf25f627ea]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x55bf25f5e1a7]
 mongod(+0xC315E1) [0x55bf25f615e1]
 mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x55bf26e64022]
 mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x55bf25f5d00f]
 mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x55bf25f5f555]
 mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x55bf25f5fe4b]
 mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x55bf25f5e22d]
 mongod(+0xC315E1) [0x55bf25f615e1]
 mongod(+0x1B34585) [0x55bf26e64585]
 mongod(+0x20F2584) [0x55bf27422584]
 libpthread.so.0(+0x76BA) [0x7f93edae76ba]
 libc.so.6(clone+0x6D) [0x7f93ed81d41d]
-----  END BACKTRACE  -----

The attached screenshot is from a process monitor showing mongodb using almost all the system memory.



 Comments   
Comment by Kelsey Schubert [ 25/Oct/18 ]

Hi travis@bryx.com,

We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Regards,
Kelsey

Comment by Travis Brown [ 11/Oct/18 ]

Hi Nic,

I apologize for the delay. The response went to my spam folder. I will turn off quiet and get the diagnostic data uploaded.

Travis

Comment by Nick Brewer [ 03/Oct/18 ]

travis@bryx.com It may be difficult to diagnose the underlying cause of this issue with the --quiet option enabled, as it would be useful to line up resource utilization to specific queries or activities on the mongod Could you restart mongod without this option, and if the out of memory exception occurs again, upload:

  • The mongod log file
  • An archive (tar or zip) of the dbpath/diagnostic.data directory
    Since this ticket is public, I've generated a secure portal where you can upload these files; information shared there is only available to MongoDB employees, and is automatically removed after a period of time.

In the meantime, if you would like to upload the current mongod log and diagnostic.data, we'd be happy to see if we can identify what's causing this level of memory utilization.

Thanks,
-Nick

Generated at Thu Feb 08 04:45:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.