Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-37439

Out of memory exception

    • Type: Icon: Bug Bug
    • Resolution: Incomplete
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 3.6.7
    • Component/s: Stability
    • Labels:
      None
    • ALL
    • Hide

      Run mongodb in a 3 server cluster for ~30 days

      Show
      Run mongodb in a 3 server cluster for ~30 days

      mongodb 3.6.7 will use nearly all the memory in the system when running as primary, and eventually crash with an out of memory exception. This machine has 32GB of RAM and is running Ubuntu 16.04.5 LTS.

       

      2018-10-03T13:38:15.588+0000 I NETWORK  [conn1553489] end connection 172.20.0.1:59480 (113 connections now open)
      2018-10-03T13:38:15.689+0000 I NETWORK  [conn1553490] end connection 10.150.1.1:55372 (113 connections now open)
      2018-10-03T13:38:17.184+0000 I NETWORK  [conn1553491] received client metadata from 172.20.0.2:35972 conn1553491: { driver: { name: "NetworkInterfaceASIO-ShardRegistry", version: "3.6.6" }, os: { type: "Linux", name: "Ubu
      ntu", architecture: "x86_64", version: "14.04" } }
      2018-10-03T13:38:17.566+0000 I NETWORK  [conn1553492] received client metadata from 172.20.0.22:54262 conn1553492: { driver: { name: "PyMongo", version: "3.5.1" }, os: { type: "Linux", name: "Ubuntu 16.04 xenial", archite
      cture: "x86_64", version: "4.17.17-x86_64-linode116" }, platform: "CPython 2.7.15.final.0" }
      2018-10-03T13:38:17.832+0000 F -        [conn290616] out of memory.
      
       0x55bf275693b1 0x55bf275689e4 0x55bf2755e341 0x55bf25d2de7e 0x55bf2701d730 0x55bf25f53e38 0x55bf25f5549b 0x55bf25f562a1 0x55bf25f627ea 0x55bf25f5e1a7 0x55bf25f615e1 0x55bf26e64022 0x55bf25f5d00f 0x55bf25f5f555 0x55bf25f5
      fe4b 0x55bf25f5e22d 0x55bf25f615e1 0x55bf26e64585 0x55bf27422584 0x7f93edae76ba 0x7f93ed81d41d
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"55BF25330000","o":"22393B1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55BF25330000","o":"22389E4","s":"_ZN5mongo29reportOutOfMemoryErrorAndExitEv"},{"b":"55BF25330000","o":"222E341","s":"_ZN5mongo12m
      ongoReallocEPvm"},{"b":"55BF25330000","o":"9FDE7E","s":"_ZN5mongo11_BufBuilderINS_21SharedBufferAllocatorEE15grow_reallocateEi"},{"b":"55BF25330000","o":"1CED730","s":"_ZN5mongo3rpc17OpMsgReplyBuilder22getInPlaceReplyBuil
      derEm"},{"b":"55BF25330000","o":"C23E38"},{"b":"55BF25330000","o":"C2549B"},{"b":"55BF25330000","o":"C262A1","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55BF2533000
      0","o":"C327EA","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55BF25330000","o":"C2E1A7","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55BF25330000","o":
      "C315E1"},{"b":"55BF25330000","o":"1B34022","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55BF25330000","o":"C2D
      00F","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55BF25330000","o":"C2F555","s":"_ZN5mo
      ngo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55BF25330000","o":"C2FE4B","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55BF25330000","o":"C2E22D","s":"_ZN5mongo19ServiceSta
      teMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55BF25330000","o":"C315E1"},{"b":"55BF25330000","o":"1B34585"},{"b":"55BF25330000","o":"20F2584"},{"b":"7F93EDAE0000","o":"76BA"},{"b":"7F93ED716000","o":"10741D","s":
      "clone"}],"processInfo":{ "mongodbVersion" : "3.6.7", "gitVersion" : "2628472127e9f1826e02c665c1d93880a204075e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.17.17-x86_64-linode116", "version" :
       "#1 SMP PREEMPT Mon Aug 20 16:07:40 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55BF25330000", "elfType" : 3, "buildId" : "5D78F445F57AA961C35C97316819BF42C1939FFF" }, { "b" : "7FFFA2BB7000", "elfType" : 3, "
      buildId" : "20B50035DDDF6D83A1FA4A5C699462D404FFB672" }, { "b" : "7F93EECD5000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "6EF73266978476EF9F2FD2CF31E57F4597CB74F8" }, { "b" : "7F93EE891
      000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "250E875F74377DFC74DE48BF80CCB237BB4EFF1D" }, { "b" : "7F93EE628000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3,
      "buildId" : "513282AC7EB386E2C0133FD9E1B6B8A0F38B047D" }, { "b" : "7F93EE424000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "8CC8D0D119B142D839800BFF71FB71E73AEA7BD4" }, { "b" : "7F93EE21C000
      ", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "89C34D7A182387D76D5CDA1F7718F5D58824DFB3" }, { "b" : "7F93EDF13000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "DFB8
      5DE42DAFFD09640C8FE377D572DE3E168920" }, { "b" : "7F93EDCFD000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "68220AE2C65D65C1B6AAA12FA6765A6EC2F5F434" }, { "b" : "7F93EDAE0000", "path" : "/
      lib/x86_64-linux-gnu/libpthread.so.0", "elfTy
      pe" : 3, "buildId" : "CE17E023542265FC11D9BC8F534BB4F070493D30" }, { "b" : "7F93ED716000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B5381A457906D279073822A5CEB24C4BFEF94DDB" }, { "b" : "7F93EEEF0000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5D7B6259552275A3C17BD4C3FD05F5A6BF40CAA5" } ] }}
       mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55bf275693b1]
       mongod(_ZN5mongo29reportOutOfMemoryErrorAndExitEv+0x84) [0x55bf275689e4]
       mongod(_ZN5mongo12mongoReallocEPvm+0x21) [0x55bf2755e341]
       mongod(_ZN5mongo11_BufBuilderINS_21SharedBufferAllocatorEE15grow_reallocateEi+0x5E) [0x55bf25d2de7e]
       mongod(_ZN5mongo3rpc17OpMsgReplyBuilder22getInPlaceReplyBuilderEm+0x80) [0x55bf2701d730]
       mongod(+0xC23E38) [0x55bf25f53e38]
       mongod(+0xC2549B) [0x55bf25f5549b]
       mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2A1) [0x55bf25f562a1]
       mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55bf25f627ea]
       mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x55bf25f5e1a7]
       mongod(+0xC315E1) [0x55bf25f615e1]
       mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x55bf26e64022]
       mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x15F) [0x55bf25f5d00f]
       mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xAF5) [0x55bf25f5f555]
       mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x23B) [0x55bf25f5fe4b]
       mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x55bf25f5e22d]
       mongod(+0xC315E1) [0x55bf25f615e1]
       mongod(+0x1B34585) [0x55bf26e64585]
       mongod(+0x20F2584) [0x55bf27422584]
       libpthread.so.0(+0x76BA) [0x7f93edae76ba]
       libc.so.6(clone+0x6D) [0x7f93ed81d41d]
      -----  END BACKTRACE  -----
      

      The attached screenshot is from a process monitor showing mongodb using almost all the system memory.

            Assignee:
            kelsey.schubert@mongodb.com Kelsey Schubert
            Reporter:
            travis@bryx.com Travis Brown
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: