[SERVER-22721] Mongo shell suffers memory overflow while printing sh.status(true) Created: 18/Feb/16  Updated: 06/Dec/22  Resolved: 09/Dec/19

Status: Closed
Project: Core Server
Component/s: Sharding
Affects Version/s: 3.0.9, 3.2.3, 3.3.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Angshuman Bagchi (Inactive) Assignee: [DO NOT USE] Backlog - Sharding Team
Resolution: Won't Fix Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Assigned Teams:
Sharding
Operating System: ALL
Participants:
Case:

 Description   

A customer has a fairly large dataset, 50 TB compressed and their config database is > 1 GB in size with 2.3 million chunks. When they try to print out sh.status(true) the 3.0.9 (v8) shell crashes:

> sh.status(true)
 
#
# Fatal error in CALL_AND_RETRY_0
# Allocation failed - process out of memory
#
 
2016-02-17T22:07:30.585-0800 F -        Got signal: 6 (Abort trap: 6).
 
 0x107682f99 0x1076829d6 0x7fff92c77f1a 0x10775c5e0 0x7fff92b209b3 0x107912339 0x107789239 0x10776a6c0 0x107742e68 0x1077e4bfc 0x10775bff2 0x107764f70 0x1075dd8f3 0x1075e0682 0x107784d84 0x16b135806362 0x16b135868e74 0x16b135868129
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"1074CA000","o":"1B8F99","s":"_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE"},{"b":"1074CA000","o":"1B89D6","s":"_ZN5mongo12_GLOBAL__N_110abruptQuitEi"},{"b":"7FFF92C73000","o":"4F1A","s":"_sigtramp"},{"b":"1074CA000","o":"2925E0","s":"_ZN2v8L26RecursivelySerializeToUtf8EPNS_8internal6StringEPciiiiPi"},{"b":"7FFF92AC3000","o":"5D9B3","s":"abort"},{"b":"1074CA000","o":"448339","s":"_ZN2v88internal2OS5AbortEv"},{"b":"1074CA000","o":"2BF239","s":"_Z9API_FatalPKcS0_z"},{"b":"1074CA000","o":"2A06C0","s":"_ZN2v8L24DefaultFatalErrorHandlerEPKcS1_"},{"b":"1074CA000","o":"278E68","s":"_ZN2v88internal2V823FatalProcessOutOfMemoryEPKcb"},{"b":"1074CA000","o":"31ABFC","s":"_ZN2v88internal13FlattenStringENS0_6HandleINS0_6StringEEE"},{"b":"1074CA000","o":"291FF2","s":"_ZNK2v86String9WriteUtf8EPciPii"},{"b":"1074CA000","o":"29AF70","s":"_ZN2v86String9Utf8ValueC2ENS_6HandleINS_5ValueEEE"},{"b":"1074CA000","o":"1138F3","s":"_ZN5mongo7V8Scope5PrintEPS0_RKN2v89ArgumentsE"},{"b":"1074CA000","o":"116682","s":"_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE"},{"b":"1074CA000","o":"2BAD84","s":"_ZN2v88internalL21Builtin_HandleApiCallENS0_12_GLOBAL__N_116BuiltinArgumentsILNS0_21BuiltinExtraArgumentsE1EEEPNS0_7IsolateE"},{"b":"0","o":"16B135806362"},{"b":"0","o":"16B135868E74"},{"b":"0","o":"16B135868129"}],"processInfo":{ "mongodbVersion" : "3.0.9", "gitVersion" : "20d60d3491908f1ae252fe452300de3978a040c7", "uname" : { "sysname" : "Darwin", "release" : "14.5.0", "version" : "Darwin Kernel Version 14.5.0: Tue Sep  1 21:23:09 PDT 2015; root:xnu-2782.50.1~1/RELEASE_X86_64", "machine" : "x86_64" }, "somap" : [ { "path" : "/usr/local/bin/mongo", "machType" : 2, "b" : "1074CA000", "vmaddr" : "100000000", "buildId" : "446D696D93283526840BE1B505990E99" }, { "path" : "/usr/lib/libSystem.B.dylib", "machType" : 6, "b" : "7FFF9322E000", "vmaddr" : "7FFF8752C000", "buildId" : "1866C519C5F33D098C17A8F703664521" }, { "path" : "/usr/lib/libc++.1.dylib", "machType" : 6, "b" : "7FFF9B430000", "vmaddr" : "7FFF8F72E000", "buildId" : "1B9530FD989B3174BB1CBDC159501710" }, { "path" : "/usr/lib/system/libcache.dylib", "machType" : 6, "b" : "7FFF9390E000", "vmaddr" : "7FFF87C0C000", "buildId" : "45E9A2E799C436B2BEE30C4E11614AD1" }, { "path" : "/usr/lib/system/libcommonCrypto.dylib", "machType" : 6, "b" : "7FFF8F60B000", "vmaddr" : "7FFF83909000", "buildId" : "E789748DF9A73CFFB31790DF348B1E95" }, { "path" : "/usr/lib/system/libcompiler_rt.dylib", "machType" : 6, "b" : "7FFF8EDDE000", "vmaddr" : "7FFF830DC000", "buildId" : "BF8FC133EE103DA69B9092039E28678F" }, { "path" : "/usr/lib/system/libcopyfile.dylib", "machType" : 6, "b" : "7FFF98AA7000", "vmaddr" : "7FFF8CDA5000", "buildId" : "0C68D3A6ACDD3EF3991ACC82C32AB836" }, { "path" : "/usr/lib/system/libcorecrypto.dylib", "machType" : 6, "b" : "7FFF949D2000", "vmaddr" : "7FFF88CD0000", "buildId" : "5779FFA04D9A3AD4B7F2618227621DC8" }, { "path" : "/usr/lib/system/libdispatch.dylib", "machType" : 6, "b" : "7FFF97C03000", "vmaddr" : "7FFF8BF01000", "buildId" : "A61E703C784A3698B51375DD12AAD6DC" }, { "path" : "/usr/lib/system/libdyld.dylib", "machType" : 6, "b" : "7FFF94AB4000", "vmaddr" : "7FFF88DB2000", "buildId" : "CFBBE540D5033AFCB5D6644F1E69949B" }, { "path" : "/usr/lib/system/libkeymgr.dylib", "machType" : 6, "b" : "7FFF9704E000", "vmaddr" : "7FFF8B34C000", "buildId" : "77845842DE703CC5BD01C3D14227CED5" }, { "path" : "/usr/lib/system/liblaunch.dylib", "machType" : 6, "b" : "7FFF944DB000", "vmaddr" : "7FFF887D9000", "buildId" : "4F81CA3AD2CE3030A89D42F3DAD7BA8F" }, { "path" : "/usr/lib/system/libmacho.dylib", "machType" : 6, "b" : "7FFF967DB000", "vmaddr" : "7FFF8AAD9000", "buildId" : "126CA2EDDE91308F8881B9DAEC3C63B6" }, { "path" : "/usr/lib/system/libquarantine.dylib", "machType" : 6, "b" : "7FFF94A78000", "vmaddr" : "7FFF88D76000", "buildId" : "7AF900412768378A925AD83161863642" }, { "path" : "/usr/lib/system/libremovefile.dylib", "machType" : 6, "b" : "7FFF91E63000", "vmaddr" : "7FFF86161000", "buildId" : "3485B5F46CE83C628DFD8736ED6E8531" }, { "path" : "/usr/lib/system/libsystem_asl.dylib", "machType" : 6, "b" : "7FFF8E48C000", "vmaddr" : "7FFF8278A000", "buildId" : "F153AC5B0542356E88C820A62CA704E2" }, { "path" : "/usr/lib/system/libsystem_blocks.dylib", "machType" : 6, "b" : "7FFF91E65000", "vmaddr" : "7FFF86163000", "buildId" : "9615D10AFCA73BE4AA1A1B195DACE1A1" }, { "path" : "/usr/lib/system/libsystem_c.dylib", "machType" : 6, "b" : "7FFF92AC3000", "vmaddr" : "7FFF86DC1000", "buildId" : "F0635E0FFE4B34DBACF9A58C1E9070E9" }, { "path" : "/usr/lib/system/libsystem_configuration.dylib", "machType" : 6, "b" : "7FFF92CD6000", "vmaddr" : "7FFF86FD4000", "buildId" : "56F94DCEDBDE36158F07DE6270D9F8BE" }, { "path" : "/usr/lib/system/libsystem_coreservices.dylib", "machType" : 6, "b" : "7FFF92BF0000", "vmaddr" : "7FFF86EEE000", "buildId" : "41B7C5785A5331C8A96FC73E030B0938" }, { "path" : "/usr/lib/system/libsystem_coretls.dylib", "machType" : 6, "b" : "7FFF8BD0A000", "vmaddr" : "7FFF80008000", "buildId" : "155DA0A92046332EBFA3D7974A51F731" }, { "path" : "/usr/lib/system/libsystem_dnssd.dylib", "machType" : 6, "b" : "7FFF93913000", "vmaddr" : "7FFF87C11000", "buildId" : "9EC5AF92D0D23BDE92B6D3730D3865C8" }, { "path" : "/usr/lib/system/libsystem_info.dylib", "machType" : 6, "b" : "7FFF8E43B000", "vmaddr" : "7FFF82739000", "buildId" : "2E16C4B3A32739579C41143911979A1E" }, { "path" : "/usr/lib/system/libsystem_kernel.dylib", "machType" : 6, "b" : "7FFF94A7B000", "vmaddr" : "7FFF88D79000", "buildId" : "1EE815DAFF1B3A53AE9BC98BD8177A9D" }, { "path" : "/usr/lib/system/libsystem_m.dylib", "machType" : 6, "b" : "7FFF8BE2C000", "vmaddr" : "7FFF8012A000", "buildId" : "1E12AB456D9636D0A226F24D9FB0D9D6" }, { "path" : "/usr/lib/system/libsystem_malloc.dylib", "machType" : 6, "b" : "7FFF92BCE000", "vmaddr" : "7FFF86ECC000", "buildId" : "DDA8928BCC0D3255BD8A3FEA0982B890" }, { "path" : "/usr/lib/system/libsystem_network.dylib", "machType" : 6, "b" : "7FFF9243D000", "vmaddr" : "7FFF8673B000", "buildId" : "6105C13467223C0AA4CE5E1261E2E1CC" }, { "path" : "/usr/lib/system/libsystem_networkextension.dylib", "machType" : 6, "b" : "7FFF93022000", "vmaddr" : "7FFF87320000", "buildId" : "BA58B30B83773B0A8AE34F84021D9D4E" }, { "path" : "/usr/lib/system/libsystem_notify.dylib", "machType" : 6, "b" : "7FFF98C5B000", "vmaddr" : "7FFF8CF59000", "buildId" : "61147800F3203DAA850CBADF33855F29" }, { "path" : "/usr/lib/system/libsystem_platform.dylib", "machType" : 6, "b" : "7FFF92C73000", "vmaddr" : "7FFF86F71000", "buildId" : "64E34079D7123D669CE2418624A5C040" }, { "path" : "/usr/lib/system/libsystem_pthread.dylib", "machType" : 6, "b" : "7FFF94646000", "vmaddr" : "7FFF88944000", "buildId" : "ACE90967ECD03251AEEB461E3C6414F7" }, { "path" : "/usr/lib/system/libsystem_sandbox.dylib", "machType" : 6, "b" : "7FFF94A4C000", "vmaddr" : "7FFF88D4A000", "buildId" : "3F5E973FC70231AC97BC05F5C195683C" }, { "path" : "/usr/lib/system/libsystem_secinit.dylib", "machType" : 6, "b" : "7FFF90475000", "vmaddr" : "7FFF84773000", "buildId" : "581DAD0F6B633A48B63B917AF799ABAA" }, { "path" : "/usr/lib/system/libsystem_stats.dylib", "machType" : 6, "b" : "7FFF8F682000", "vmaddr" : "7FFF83980000", "buildId" : "D0E968373CF6323DB7116DF6F660E530" }, { "path" : "/usr/lib/system/libsystem_trace.dylib", "machType" : 6, "b" : "7FFF8DB00000", "vmaddr" : "7FFF81DFE000", "buildId" : "840F5301B55A307890B9FEFFD6CD741A" }, { "path" : "/usr/lib/system/libunc.dylib", "machType" : 6, "b" : "7FFF90BED000", "vmaddr" : "7FFF84EEB000", "buildId" : "5676F7EAC1DF329FB006D2C3022B7D70" }, { "path" : "/usr/lib/system/libunwind.dylib", "machType" : 6, "b" : "7FFF92CAD000", "vmaddr" : "7FFF86FAB000", "buildId" : "BE7E51A0B6EA3A549CCA9D88F683A6D6" }, { "path" : "/usr/lib/system/libxpc.dylib", "machType" : 6, "b" : "7FFF935BA000", "vmaddr" : "7FFF878B8000", "buildId" : "5C829202962E37448B5000D38CC88E84" }, { "path" : "/usr/lib/libobjc.A.dylib", "machType" : 6, "b" : "7FFF993D8000", "vmaddr" : "7FFF8D6D6000", "buildId" : "759E155DBC423D4E869B6F57D477177C" }, { "path" : "/usr/lib/libauto.dylib", "machType" : 6, "b" : "7FFF90DE4000", "vmaddr" : "7FFF850E2000", "buildId" : "A260789BD4D8316A9490254767B8A5F1" }, { "path" : "/usr/lib/libc++abi.dylib", "machType" : 6, "b" : "7FFF9676B000", "vmaddr" : "7FFF8AA69000", "buildId" : "88A22A0F87C63002BFBAAC0F2808B8B9" }, { "path" : "/usr/lib/libDiagnosticMessagesClient.dylib", "machType" : 6, "b" : "7FFF9B940000", "vmaddr" : "7FFF8FC3E000", "buildId" : "2EE8E4365CDC34C599595BA218D507FB" } ] }}
 mongo(_ZN5mongo15printStackTraceERNSt3__113basic_ostreamIcNS0_11char_traitsIcEEEE+0x39) [0x107682f99]
 mongo(_ZN5mongo12_GLOBAL__N_110abruptQuitEi+0xB6) [0x1076829d6]
 libsystem_platform.dylib(_sigtramp+0x1A) [0x7fff92c77f1a]
 mongo(_ZN2v8L26RecursivelySerializeToUtf8EPNS_8internal6StringEPciiiiPi+0xF0) [0x10775c5e0]
 libsystem_c.dylib(abort+0x81) [0x7fff92b209b3]
 mongo(_ZN2v88internal2OS5AbortEv+0x9) [0x107912339]
 mongo(_Z9API_FatalPKcS0_z+0xB9) [0x107789239]
 mongo(_ZN2v8L24DefaultFatalErrorHandlerEPKcS1_+0x70) [0x10776a6c0]
 mongo(_ZN2v88internal2V823FatalProcessOutOfMemoryEPKcb+0x218) [0x107742e68]
 mongo(_ZN2v88internal13FlattenStringENS0_6HandleINS0_6StringEEE+0x1EC) [0x1077e4bfc]
 mongo(_ZNK2v86String9WriteUtf8EPciPii+0x2A2) [0x10775bff2]
 mongo(_ZN2v86String9Utf8ValueC2ENS_6HandleINS_5ValueEEE+0x170) [0x107764f70]
 mongo(_ZN5mongo7V8Scope5PrintEPS0_RKN2v89ArgumentsE+0x133) [0x1075dd8f3]
 mongo(_ZN5mongo7V8Scope10v8CallbackERKN2v89ArgumentsE+0x52) [0x1075e0682]
 mongo(_ZN2v88internalL21Builtin_HandleApiCallENS0_12_GLOBAL__N_116BuiltinArgumentsILNS0_21BuiltinExtraArgumentsE1EEEPNS0_7IsolateE+0x1F4) [0x107784d84]
 ??? [0x16b135806362]
 ??? [0x16b135868e74]
 ??? [0x16b135868129]
-----  END BACKTRACE  -----

The 3.2.3 (spider monkey) shell handles this a little better, but not optimally:

angshuman-macbook:CS-27259 angshuman.bagchi$ mongo --port 32000
MongoDB shell version: 3.2.3
connecting to: 127.0.0.1:32000/test
Server has startup warnings: 
2016-02-17T21:59:36.649-0800 I CONTROL  [initandlisten] 
2016-02-17T21:59:36.650-0800 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
> sh.status(true)
Cannot use 'commands' readMode, degrading to 'legacy' mode
2016-02-17T22:14:52.908-0800 E QUERY    [thread1] InternalError: allocation size overflow :
printShardingStatus/output@src/mongo/shell/utils_sh.js:542:9
printShardingStatus/</</<@src/mongo/shell/utils_sh.js:721:1
DBQuery.prototype.forEach@src/mongo/shell/query.js:477:1
printShardingStatus/</<@src/mongo/shell/utils_sh.js:719:33
DBQuery.prototype.forEach@src/mongo/shell/query.js:477:1
printShardingStatus/<@src/mongo/shell/utils_sh.js:693:1
DBQuery.prototype.forEach@src/mongo/shell/query.js:477:1
printShardingStatus@src/mongo/shell/utils_sh.js:674:1
sh.status@src/mongo/shell/utils_sh.js:65:5
@(shell):1:1

It does not crash, but does not display any useful information either.



 Comments   
Comment by Kevin Pulo [ 01/Nov/18 ]

Ideally this should be done in a way that also allows programmatic access by scripts. For example, have sh.status() return an object that implements shellPrint() to interactively output the existing/legacy format, but also provides functions to separately print (or optionally convert to string) each "section" of output.

More importantly, there could be accessors for each bit of info in the output (natively, ie. ideally as a cursor, and definitely not as a string). This would allow doing things like sh.status().chunks() to get a cursor over the chunks, .collections() to get a cursor on the collections, etc. Since these would basically be sugar over the appropriate find() command, they could include passing query predicates/projections, and you could append stuff like .sort({min:1}).limit(1) to find the first chunk, and so on. The output routines could also use these accessors to properly iterate over the metadata when generating the output.

printShardingStatus() would then need to be updated to do sh.status().shellPrint(). But this would also allow the annoyingly inconsistent args to sh.status() vs printShardingStatus() to be fixed, eg. sh.status().configDB(db.getSiblingDB("dump")).verbose() instead of sh.status(true, db.getSiblingDB("dump")) or printShardingStatus(db.getSiblingDB("dump"), true).

Comment by Kaloian Manassiev [ 18/Feb/16 ]

The problem is that the way the sh.status function is written it concatenates everything in one string and prints it all at once afterwards.

Given that this function is run interactively from the shell it is very unlikely that something else will be printed in between to break the output, so we should be able to make it just print for now.

Generated at Thu Feb 08 04:01:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.