Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-22721

Mongo shell suffers memory overflow while printing sh.status(true)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Won't Fix
    • Affects Version/s: 3.0.9, 3.2.3, 3.3.1
    • Fix Version/s: None
    • Component/s: Sharding
    • Labels:
      None
    • Operating System:
      ALL
    • Case:

      Description

      A customer has a fairly large dataset, 50 TB compressed and their config database is > 1 GB in size with 2.3 million chunks. When they try to print out sh.status(true) the 3.0.9 (v8) shell crashes:

      > sh.status(true)
       
      #
      # Fatal error in CALL_AND_RETRY_0
      # Allocation failed - process out of memory
      #
       
      2016-02-17T22:07:30.585-0800 F -        Got signal: 6 (Abort trap: 6).
       
       0x107682f99 0x1076829d6 0x7fff92c77f1a 0x10775c5e0 0x7fff92b209b3 0x107912339 0x107789239 0x10776a6c0 0x107742e68 0x1077e4bfc 0x10775bff2 0x107764f70 0x1075dd8f3 0x1075e0682 0x107784d84 0x16b135806362 0x16b135868e74 0x16b135868129
      ----- BEGIN BACKTRACE -----
      {"backtrace":[{"b":"1074CA000","o":"1B8F99","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"1074CA000","o":"1B89D6","s":"_ZN5mongo12_GLOBAL__N_110abruptQuitEi"},{"b":"7FFF92C73000","o":"4F1A","s":"_sigtramp"},{"b":"1074CA000","o":"2925E0","s":"_ZN2v8L26RecursivelySerializeToUtf8EPNS_8internal6StringEPciiiiPi"},{"b":"7FFF92AC3000","o":"5D9B3","s":"abort"},{"b":"1074CA000","o":"448339","s":"_ZN2v88internal2OS5AbortEv"},{"b":"1074CA000","o":"2BF239","s":"_Z9API_FatalPKcS0_z"},{"b":"1074CA000","o":"2A06C0","s":"_ZN2v8L24DefaultFatalErrorHandlerEPKcS1_"},{"b":"1074CA000","o":"278E68","s":"_ZN2v88internal2V823FatalProcessOutOfMemoryEPKcb"},{"b":"1074CA000","o":"31ABFC","s":"_ZN2v88internal13FlattenStringENS0_6HandleINS0_6StringEEE"},{"b":"1074CA000","o":"291FF2","s":"_ZNK2v86String9WriteUtf8EPciPii"},{"b":"1074CA000","o":"29AF70","s":"_ZN2v86String9Utf8ValueC2ENS_6HandleINS_5ValueEEE"},{"b":"1074CA000","o":"1138F3","s":"_ZN5mongo7V8Scope5PrintEPS0_RKN2v89ArgumentsE"},{"b":"1074CA000","o":"116682","s":"_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE"},{"b":"1074CA000","o":"2BAD84","s":"_ZN2v88internalL21Builtin_HandleApiCallENS0_12_GLOBAL__N_116BuiltinArgumentsILNS0_21BuiltinExtraArgumentsE1EEEPNS0_7IsolateE"},{"b":"0","o":"16B135806362"},{"b":"0","o":"16B135868E74"},{"b":"0","o":"16B135868129"}],"processInfo":{ "mongodbVersion" : "3.0.9", "gitVersion" : "20d60d3491908f1ae252fe452300de3978a040c7", "uname" : { "sysname" : "Darwin", "release" : "14.5.0", "version" : "Darwin Kernel Version 14.5.0: Tue Sep  1 21:23:09 PDT 2015; root:xnu-2782.50.1~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/usr/local/bin/mongo", "machType" : 2, "b" : "1074CA000", "vmaddr" : "100000000", "buildId" : "446D696D93283526840BE1B505990E99" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF9322E000", "vmaddr" : "7FFF8752C000", "buildId" : "1866C519C5F33D098C17A8F703664521" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF9B430000", "vmaddr" : "7FFF8F72E000", "buildId" : "1B9530FD989B3174BB1CBDC159501710" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF9390E000", "vmaddr" : "7FFF87C0C000", "buildId" : "45E9A2E799C436B2BEE30C4E11614AD1" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8F60B000", "vmaddr" : "7FFF83909000", "buildId" : "E789748DF9A73CFFB31790DF348B1E95" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF8EDDE000", "vmaddr" : "7FFF830DC000", "buildId" : "BF8FC133EE103DA69B9092039E28678F" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF98AA7000", "vmaddr" : "7FFF8CDA5000", "buildId" : "0C68D3A6ACDD3EF3991ACC82C32AB836" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF949D2000", "vmaddr" : "7FFF88CD0000", "buildId" : "5779FFA04D9A3AD4B7F2618227621DC8" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF97C03000", "vmaddr" : "7FFF8BF01000", "buildId" : "A61E703C784A3698B51375DD12AAD6DC" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF94AB4000", "vmaddr" : "7FFF88DB2000", "buildId" : "CFBBE540D5033AFCB5D6644F1E69949B" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF9704E000", "vmaddr" : "7FFF8B34C000", "buildId" : "77845842DE703CC5BD01C3D14227CED5" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF944DB000", "vmaddr" : "7FFF887D9000", "buildId" : "4F81CA3AD2CE3030A89D42F3DAD7BA8F" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF967DB000", "vmaddr" : "7FFF8AAD9000", "buildId" : "126CA2EDDE91308F8881B9DAEC3C63B6" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF94A78000", "vmaddr" : "7FFF88D76000", "buildId" : "7AF900412768378A925AD83161863642" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF91E63000", "vmaddr" : "7FFF86161000", "buildId" : "3485B5F46CE83C628DFD8736ED6E8531" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF8E48C000", "vmaddr" : "7FFF8278A000", "buildId" : "F153AC5B0542356E88C820A62CA704E2" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF91E65000", "vmaddr" : "7FFF86163000", "buildId" : "9615D10AFCA73BE4AA1A1B195DACE1A1" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF92AC3000", "vmaddr" : "7FFF86DC1000", "buildId" : "F0635E0FFE4B34DBACF9A58C1E9070E9" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF92CD6000", "vmaddr" : "7FFF86FD4000", "buildId" : "56F94DCEDBDE36158F07DE6270D9F8BE" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF92BF0000", "vmaddr" : "7FFF86EEE000", "buildId" : "41B7C5785A5331C8A96FC73E030B0938" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF8BD0A000", "vmaddr" : "7FFF80008000", "buildId" : "155DA0A92046332EBFA3D7974A51F731" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF93913000", "vmaddr" : "7FFF87C11000", "buildId" : "9EC5AF92D0D23BDE92B6D3730D3865C8" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF8E43B000", "vmaddr" : "7FFF82739000", "buildId" : "2E16C4B3A32739579C41143911979A1E" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF94A7B000", "vmaddr" : "7FFF88D79000", "buildId" : "1EE815DAFF1B3A53AE9BC98BD8177A9D" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8BE2C000", "vmaddr" : "7FFF8012A000", "buildId" : "1E12AB456D9636D0A226F24D9FB0D9D6" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF92BCE000", "vmaddr" : "7FFF86ECC000", "buildId" : "DDA8928BCC0D3255BD8A3FEA0982B890" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF9243D000", "vmaddr" : "7FFF8673B000", "buildId" : "6105C13467223C0AA4CE5E1261E2E1CC" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF93022000", "vmaddr" : "7FFF87320000", "buildId" : "BA58B30B83773B0A8AE34F84021D9D4E" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF98C5B000", "vmaddr" : "7FFF8CF59000", "buildId" : "61147800F3203DAA850CBADF33855F29" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF92C73000", "vmaddr" : "7FFF86F71000", "buildId" : "64E34079D7123D669CE2418624A5C040" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF94646000", "vmaddr" : "7FFF88944000", "buildId" : "ACE90967ECD03251AEEB461E3C6414F7" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF94A4C000", "vmaddr" : "7FFF88D4A000", "buildId" : "3F5E973FC70231AC97BC05F5C195683C" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF90475000", "vmaddr" : "7FFF84773000", "buildId" : "581DAD0F6B633A48B63B917AF799ABAA" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF8F682000", "vmaddr" : "7FFF83980000", "buildId" : "D0E968373CF6323DB7116DF6F660E530" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF8DB00000", "vmaddr" : "7FFF81DFE000", "buildId" : "840F5301B55A307890B9FEFFD6CD741A" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF90BED000", "vmaddr" : "7FFF84EEB000", "buildId" : "5676F7EAC1DF329FB006D2C3022B7D70" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF92CAD000", "vmaddr" : "7FFF86FAB000", "buildId" : "BE7E51A0B6EA3A549CCA9D88F683A6D6" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF935BA000", "vmaddr" : "7FFF878B8000", "buildId" : "5C829202962E37448B5000D38CC88E84" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF993D8000", "vmaddr" : "7FFF8D6D6000", "buildId" : "759E155DBC423D4E869B6F57D477177C" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF90DE4000", "vmaddr" : "7FFF850E2000", "buildId" : "A260789BD4D8316A9490254767B8A5F1" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF9676B000", "vmaddr" : "7FFF8AA69000", "buildId" : "88A22A0F87C63002BFBAAC0F2808B8B9" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF9B940000", "vmaddr" : "7FFF8FC3E000", "buildId" : "2EE8E4365CDC34C599595BA218D507FB" } ] }}
       mongo(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x107682f99]
       mongo(_ZN5mongo12_GLOBAL__N_110abruptQuitEi+0xB6) [0x1076829d6]
       libsystem_platform.dylib(_sigtramp+0x1A) [0x7fff92c77f1a]
       mongo(_ZN2v8L26RecursivelySerializeToUtf8EPNS_8internal6StringEPciiiiPi+0xF0) [0x10775c5e0]
       libsystem_c.dylib(abort+0x81) [0x7fff92b209b3]
       mongo(_ZN2v88internal2OS5AbortEv+0x9) [0x107912339]
       mongo(_Z9API_FatalPKcS0_z+0xB9) [0x107789239]
       mongo(_ZN2v8L24DefaultFatalErrorHandlerEPKcS1_+0x70) [0x10776a6c0]
       mongo(_ZN2v88internal2V823FatalProcessOutOfMemoryEPKcb+0x218) [0x107742e68]
       mongo(_ZN2v88internal13FlattenStringENS0_6HandleINS0_6StringEEE+0x1EC) [0x1077e4bfc]
       mongo(_ZNK2v86String9WriteUtf8EPciPii+0x2A2) [0x10775bff2]
       mongo(_ZN2v86String9Utf8ValueC2ENS_6HandleINS_5ValueEEE+0x170) [0x107764f70]
       mongo(_ZN5mongo7V8Scope5PrintEPS0_RKN2v89ArgumentsE+0x133) [0x1075dd8f3]
       mongo(_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE+0x52) [0x1075e0682]
       mongo(_ZN2v88internalL21Builtin_HandleApiCallENS0_12_GLOBAL__N_116BuiltinArgumentsILNS0_21BuiltinExtraArgumentsE1EEEPNS0_7IsolateE+0x1F4) [0x107784d84]
       ??? [0x16b135806362]
       ??? [0x16b135868e74]
       ??? [0x16b135868129]
      -----  END BACKTRACE  -----
      

      The 3.2.3 (spider monkey) shell handles this a little better, but not optimally:

      angshuman-macbook:CS-27259 angshuman.bagchi$ mongo --port 32000
      MongoDB shell version: 3.2.3
      connecting to: 127.0.0.1:32000/test
      Server has startup warnings: 
      2016-02-17T21:59:36.649-0800 I CONTROL  [initandlisten] 
      2016-02-17T21:59:36.650-0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
      > sh.status(true)
      Cannot use 'commands' readMode, degrading to 'legacy' mode
      2016-02-17T22:14:52.908-0800 E QUERY    [thread1] InternalError: allocation size overflow :
      printShardingStatus/output@src/mongo/shell/utils_sh.js:542:9
      printShardingStatus/</</<@src/mongo/shell/utils_sh.js:721:1
      DBQuery.prototype.forEach@src/mongo/shell/query.js:477:1
      printShardingStatus/</<@src/mongo/shell/utils_sh.js:719:33
      DBQuery.prototype.forEach@src/mongo/shell/query.js:477:1
      printShardingStatus/<@src/mongo/shell/utils_sh.js:693:1
      DBQuery.prototype.forEach@src/mongo/shell/query.js:477:1
      printShardingStatus@src/mongo/shell/utils_sh.js:674:1
      sh.status@src/mongo/shell/utils_sh.js:65:5
      @(shell):1:1
      

      It does not crash, but does not display any useful information either.

        Attachments

          Activity

            People

            Assignee:
            backlog-server-sharding Backlog - Sharding Team
            Reporter:
            angshuman.bagchi Angshuman Bagchi
            Participants:
            Votes:
            2 Vote for this issue
            Watchers:
            13 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: