-
Type:
Improvement
-
Resolution: Done
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Diagnostics, Internal Code
-
Labels:None
-
(copied to CRM)
Currently, stacktrace output from mongosymb.py doesn't include the paths/filenames of any binaries. This is a problem because there are times when this information is relevant and important, but is missing from the main symbolised stacktrace that is being used diagnostically. Instead there is typically a line with no value like one of:
??:0:0: ?? ??:0:0: __libc_start_main
This is because:
- line 168 is never reached, because:
- frame["path"] always has a value (since it defaults to the mongod filename), and so always gets resolved on line 105, and:
- either debug symbols are not available for the library file in question, or the addr is inappropriate for the mongod binary (which is the default frame["path"]).
In addition, the stacktrace printing code does not output Dl_info::dli_fname (the filename of the binary for that frame) in the JSON backtrace (even though it outputs the basename of this in the "classic" output), opting instead to output dl_phdr_info::dlpi_name in the somap. The Linux man page for dl_iterate_phdr(3) says:
The first object visited by callback is the main program. For the main program, the dlpi_name field will be an empty string.
However, this is not always true — for example, LD_PRELOAD causes dlpi_name to be null/empty. In these cases, dli_fname still contains the correct filename for the frame. In the example below see the mongod_handle_request frame, which is actually in imperva_mongo_lean3.so, but this library is not mentioned (by name) in the somap.
There is also some weird relocation thing going on (ASR?), because the addr bases for symbols in libs like libc and libpthread don't match the bases reported in the somap. This further compounds the symbol resolution problems above, and is another reason to include dli_fname for each frame. In the example below, the libc and libpthread frames report addresses starting with 0x7fa6a..., whereas in the somap they have base addresses starting with 0x7f6b7...:
| lib | libc | libpthread |
|---|---|---|
| frame base addr | 7FA6A6C95000 | 7FA6A7029000 |
| somap base addr | 7F6B7D495000 | 7F6B7CC29000 |
The specific changes I'd like to suggest are:
- Include either dlinfo.dli_fname or getBaseName(dlinfo.dli_fname) for each frame in the JSON backtrace object (as a new field in the backtrace array sub-objects).
- Currently only the basename is logged, not the full path. Logging the full path would leak the filesystem location where the mongod server has been installed, which some people may consider sensitive.
- On the other hand, the full path to the loaded binary may contain useful diagnostic information.
- Adjust mongosymb.py to output a more useful line when the symbol for a frame cannot be resolved, including at least the basename of dlinfo.dli_fname, and dlinfo.dli_sname (which is already present as the "s" field of the JSON stacktrace).
- Possibly even include the basename of dlinfo.dli_fname at the start of every line for consistency?
Example:
2019-03-01T19:48:51.768-0500 F - [conn548] Invalid access at address: 0
2019-03-01T19:48:51.793-0500 F - [conn548] Got signal: 11 (Segmentation fault).
0x557310dc4721 0x557310dc3939 0x557310dc3fa6 0x7fa6a70387e0 0x557310eb1413 0x557310eb14ec 0x557310f53dda 0x55730fb3c6cc 0x55730fb3caa7 0x55730f67ebbc 0x55730f663670 0x55730f65d118 0x55731070406f 0x55730f5a35c4 0x55730f5a4be7 0x55730f5a5a44 0x7fa68faf19dc 0x55730f5b3aaa 0x55730f5af407 0x55730f5b2891 0x5573106eede2 0x55730f5ae240 0x55730f5b07d5 0x55730f5b10d1 0x55730f5af48d 0x55730f5b2891 0x5573106ef345 0x557310c43e94 0x7fa6a7030aa1 0x7fa6a6d7dbdd
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55730E9EC000","o":"23D8721","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55730E9EC000","o":"23D7939"},{"b":"55730E9EC000","o":"23D7FA6"},{"b":"7FA6A7029000","o":"F7E0"},{"b":"55730E9EC000","o":"24C5413","s":"_ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEmi"},{"b":"55730E9EC000","o":"24C54EC","s":"_ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEm"},{"b":"55730E9EC000","o":"2567DDA","s":"_ZdlPvRKSt9nothrow_t"},{"b":"55730E9EC000","o":"11506CC","s":"_ZN5mongo12PlanExecutorD1Ev"},{"b":"55730E9EC000","o":"1150AA7","s":"_ZN5mongo12PlanExecutor7DeleterclEPS0_"},{"b":"55730E9EC000","o":"C92BBC","s":"_ZN5mongo14performUpdatesEPNS_16OperationContextERKNS_9write_ops6UpdateE"},{"b":"55730E9EC000","o":"C77670"},{"b":"55730E9EC000","o":"C71118"},{"b":"55730E9EC000","o":"1D1806F","s":"_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE"},{"b":"55730E9EC000","o":"BB75C4"},{"b":"55730E9EC000","o":"BB8BE7"},{"b":"55730E9EC000","o":"BB9A44","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"7FA68FA83000","o":"6E9DC","s":"mongod_handle_request"},{"b":"55730E9EC000","o":"BC7AAA","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55730E9EC000","o":"BC3407","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55730E9EC000","o":"BC6891"},{"b":"55730E9EC000","o":"1D02DE2","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55730E9EC000","o":"BC2240","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55730E9EC000","o":"BC47D5","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55730E9EC000","o":"BC50D1","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55730E9EC000","o":"BC348D","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55730E9EC000","o":"BC6891"},{"b":"55730E9EC000","o":"1D03345"},{"b":"55730E9EC000","o":"2257E94"},{"b":"7FA6A7029000","o":"7AA1"},{"b":"7FA6A6C95000","o":"E8BDD","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.9", "gitVersion" : "167861a164723168adfaaa866f310cb94010428f", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "2.6.32-754.9.1.el6.x86_64", "version" : "#1 SMP Wed Nov 21 15:08:21 EST 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55730E9EC000", "elfType" : 3, "buildId" : "5FA5D557062B7FE04CE2E5DC5FC9721363990AD7" }, { "b" : "7FFF02038000", "elfType" : 3, "buildId" : "BECEDAD58AF447BF6547467F40C2659CA3F97AC0" }, { "b" : "7F74F4E2A000", "path" : "/usr/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "A38B9CE8AEAF277CBD8BC1298B1731E2C9A66192" }, { "b" : "7FA6A95DE000", "path" : "/usr/lib64/libnetsnmpagent.so.20", "elfType" : 3, "buildId" : "1270BB069D761BD79C79F8986BB3ED5DCAA7D06D" }, { "b" : "7FA6A93B8000", "path" : "/usr/lib64/libnetsnmphelpers.so.20", "elfType" : 3, "buildId" : "3FA4F246A7DF00EC1355C5226C9308DC7B4AB5CD" }, { "b" : "7FA6A8EF0000", "path" : "/usr/lib64/libnetsnmpmibs.so.20", "elfType" : 3, "buildId" : "AE65092368DDB948A32B62D613DD8FFE210EBEB9" }, { "b" : "7F6B7DC15000", "path" : "/usr/lib64/libnetsnmp.so.20", "elfType" : 3, "buildId" : "52E4D411A95E6C7FCCE0E1942B525AC8FBBDF4A8" }, { "b" : "7F6B76DFB000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "E0AEE889D5BF1373F2F9EE0D448DBF3F5B5113F0" }, { "b" : "7F74F51AA000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "DDBAC283102A61D6A63B3F3952A1C06657FF3AE8" }, { "b" : "7F74F539B000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "244D2593BDE4FE657BC88572DB5DA88FA274B7F3" }, { "b" : "7F6B74F57000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "0C249DF4D77989253CCD859956BF50749308A16A" }, { "b" : "7F6B7C93D000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F0BE1166EDCFFB2422B940D601A1BBD89352D80F" }, { "b" : "7F6B79558000", "path" : "/usr/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "1EDB45C205A844A75EBBB4F0075E705803FFB85B" }, { "b" : "7F6B71AEC000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "D256E285C5E11D9A99EB04CA7651003A8F67B64E" }, { "b" : "7F6B7D8E8000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "1F7E85410384392BC51FA7324961719A10125F31" }, { "b" : "7F6B7CEE0000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "FDF3A36FFFE08375456D59DA959EAB2FC30B6186" }, { "b" : "7F6B7D85C000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "8A852AC42F0B64F0F30C760EBBCFA3FE4A228F12" }, { "b" : "7F6B7B646000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDC925E58FE28DCA536993EB13179C739F1E6566" }, { "b" : "7F6B7CC29000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "85104ECFE42C606B31C2D0D0D2E5DACD3286A341" }, { "b" : "7F6B7D495000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8E3AACE76351B6A83390CA065E904EB82FBD1EC7" }, { "b" : "7F6B8067F000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1CC2165E019D43F71FDE0A47AF9F4C8EB5E51963" }, { "b" : "7F6B7AA63000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "5659EB985475B586E3BBCB95BA21F4A30BE5EBF4" }, { "b" : "7F6B7437C000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "624C7056B8BBE6BA758DEF557F516FBDBD01E1FD" }, { "b" : "7F6B73950000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "C81673692EEF670BC951EE726490F5D1CAB822F4" }, { "b" : "7F6B7534C000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "57F77704A7F1F4E3689D028D3F9ADD4E77486EC9" }, { "b" : "7F6B7B536000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "D053BB4FF0C2FC983842F81598813B9B931AD0D1" }, { "b" : "7F74F34E3000", "path" : "/usr/lib64/libssl3.so", "elfType" : 3, "buildId" : "BEF0080919EBF8ADB8C668B38BE1B19FD1F584BC" }, { "b" : "7F74F2EB6000", "path" : "/usr/lib64/libsmime3.so", "elfType" : 3, "buildId" : "2E107F018AF1E0916BE98EB24BFFE1C28CACB7F7" }, { "b" : "7F74F336E000", "path" : "/usr/lib64/libnss3.so", "elfType" : 3, "buildId" : "788E6C43CEF0FE8A2EFEF8FFD7B9B90B28EF44ED" }, { "b" : "7F6B7933F000", "path" : "/usr/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "3F385A7A46BD81EF0ED4CBCBEADC4D33AE6247E4" }, { "b" : "7F6B77D3B000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "04D1AC5F1C6C1B1AD5962DCA1225236B0B4953CE" }, { "b" : "7F6B77F36000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "C0559AF61C8808D4FC8B97D07EFFA459BFD93003" }, { "b" : "7F6B780F7000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "828BFCA03E208DFB48C4B874D81140EFC51D33C2" }, { "b" : "7F6B6EACF000", "path" : "/usr/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "8727EC925D6D91DAC74A99BDE8B3C6EE96AF13EA" }, { "b" : "7FA6A4CC4000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "083332F88CF3C61AB0184D8F397FC8BFF4548D8E" }, { "b" : "7F6B76559000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "53842C2896DED0063E1BE5C650CE97C67AE97973" }, { "b" : "7F6B77740000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "D233CCCC987214EE5DACCF88949E31469228F6FF" }, { "b" : "7F6B76909000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "F542C8ACD4AD1F2C6A551043BDFBAB051905DA1C" }, { "b" : "7F6B78306000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "2963FF1BBF4BF9131097982EB8BE5C905A342CBD" }, { "b" : "7F74F029A000", "path" : "/usr/lib64/librpm.so.1", "elfType" : 3, "buildId" : "EF9F29119A5A42A613A040DE49BE7B1B46256B21" }, { "b" : "7F74F046B000", "path" : "/usr/lib64/librpmio.so.1", "elfType" : 3, "buildId" : "DA04B3F461614CD7524D92152540F6B8303C9F45" }, { "b" : "7F6B75862000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "E7B49911F1136073DD7DC58E8118CD9A4FBE2A19" }, { "b" : "7F6B79252000", "path" : "/usr/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "6855E5BF5B3634C15F01B1043BD892D727EE3C08" }, { "b" : "7F6B71047000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "03B69EEB8998AC9CA7519A27571BAD976BA4C56D" }, { "b" : "7F6B72A44000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "3BCCABE75DC61BBA81AAE45D164E26EF4F9F55DB" }, { "b" : "7F6B76841000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "AFF1C795A3CF422C9F8AC32C7522F6376B1EA087" }, { "b" : "7F6B72E30000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "1250B1D041DD7552F0C870BB188DC3A34DF2651D" }, { "b" : "7F6B7441A000", "path" : "/usr/lib64/libelf.so.1", "elfType" : 3, "buildId" : "50517407A07B8D6C9A55A392E99246B52E8BFEEA" }, { "b" : "7F6B73DF9000", "path" : "/usr/lib64/liblzma.so.0", "elfType" : 3, "buildId" : "2F1F98636D83908F9157858BCC7B44A6A6784385" }, { "b" : "7F6B70BCC000", "path" : "/usr/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "6BDB4E1990D6EBA12A5C8D39A7650DB8798BF568" }, { "b" : "7F6B779AD000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "B4576BE308DDCF7BC31F7304E4734C3D846D0236" }, { "b" : "7F6B713A9000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "A436538388F1F25113FDA834CA2EED524EFA17D6" }, { "b" : "7F6B70DA1000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "89FDBC49F99100E42556B961E8CF1254D69E370E" }, { "b" : "7F6B7262C000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "D91C702275E2039E98E39925B02FF5C53A6C3312" }, { "b" : "7F6B72027000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8EF0683858704EF173AB11B1E27076F37F82B7B6" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x557310dc4721]
mongod(+0x23D7939) [0x557310dc3939]
mongod(+0x23D7FA6) [0x557310dc3fa6]
libpthread.so.0(+0xF7E0) [0x7fa6a70387e0]
mongod(_ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEmi+0xE3) [0x557310eb1413]
mongod(_ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEm+0x1C) [0x557310eb14ec]
mongod(_ZdlPvRKSt9nothrow_t+0x26A) [0x557310f53dda]
mongod(_ZN5mongo12PlanExecutorD1Ev+0x16C) [0x55730fb3c6cc]
mongod(_ZN5mongo12PlanExecutor7DeleterclEPS0_+0x27) [0x55730fb3caa7]
mongod(_ZN5mongo14performUpdatesEPNS_16OperationContextERKNS_9write_ops6UpdateE+0xA2C) [0x55730f67ebbc]
mongod(+0xC77670) [0x55730f663670]
mongod(+0xC71118) [0x55730f65d118]
mongod(_ZN5mongo7Command9publicRunEPNS_16OperationContextERKNS_12OpMsgRequestERNS_14BSONObjBuilderE+0x1F) [0x55731070406f]
mongod(+0xBB75C4) [0x55730f5a35c4]
mongod(+0xBB8BE7) [0x55730f5a4be7]
mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x2B4) [0x55730f5a5a44]
imperva_mongo_lean3.so(mongod_handle_request+0x5C) [0x7fa68faf19dc]
mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xBA) [0x55730f5b3aaa]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x97) [0x55730f5af407]
mongod(+0xBC6891) [0x55730f5b2891]
mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x1A2) [0x5573106eede2]
mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x150) [0x55730f5ae240]
mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xB05) [0x55730f5b07d5]
mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x241) [0x55730f5b10d1]
mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x11D) [0x55730f5af48d]
mongod(+0xBC6891) [0x55730f5b2891]
mongod(+0x1D03345) [0x5573106ef345]
mongod(+0x2257E94) [0x557310c43e94]
libpthread.so.0(+0x7AA1) [0x7fa6a7030aa1]
libc.so.6(clone+0x6D) [0x7fa6a6d7dbdd]
----- END BACKTRACE -----
{
"edition": "enterprise",
"uname": {
"release": "2.6.32-754.9.1.el6.x86_64",
"sysname": "Linux",
"machine": "x86_64",
"version": "#1 SMP Wed Nov 21 15:08:21 EST 2018"
},
"version": "3.6.9",
"buildId": "5fa5d557062b7fe04ce2e5dc5fc9721363990ad7",
"githash": "167861a164723168adfaaa866f310cb94010428f"
}
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/util/stacktrace_posix.cpp:175:0: mongo::printStackTrace(std::ostream&)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/util/signal_handlers_synchronous.cpp:182:0: mongo::(anonymous namespace)::printSignalAndBacktrace(int)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/util/signal_handlers_synchronous.cpp:278:0: mongo::(anonymous namespace)::abruptQuitWithAddrSignal(int, siginfo*, void*)
??:0:0: __libc_start_main
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/linked_list.h:75:0: tcmalloc::SLL_PopRange(void**, int, void**, void**)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/thread_cache.h:225:0: tcmalloc::ThreadCache::FreeList::PopRange(int, void**, void**)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/thread_cache.cc:195:0: tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/thread_cache.cc:157:0: tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/thread_cache.h:393:0: tcmalloc::ThreadCache::Deallocate(void*, unsigned long)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/tcmalloc.cc:1383:0: do_free_helper
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/tcmalloc.cc:1415:0: do_free_with_callback
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/tcmalloc.cc:1423:0: do_free
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/tcmalloc.cc:1688:0: operator delete(void*)
/opt/mongodbtoolchain/stow/gcc-v2.pOL/include/c++/5.4.0/bits/unique_ptr.h:76:0: std::default_delete<mongo::PlanYieldPolicy>::operator()(mongo::PlanYieldPolicy*) const
/opt/mongodbtoolchain/stow/gcc-v2.pOL/include/c++/5.4.0/bits/unique_ptr.h:236:0: std::unique_ptr<mongo::PlanYieldPolicy, std::default_delete<mongo::PlanYieldPolicy> >::~unique_ptr()
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/query/plan_executor.cpp:281:0: mongo::PlanExecutor::~PlanExecutor()
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/query/plan_executor.cpp:739:0: mongo::PlanExecutor::Deleter::operator()(mongo::PlanExecutor*)
/opt/mongodbtoolchain/stow/gcc-v2.pOL/include/c++/5.4.0/bits/unique_ptr.h:236:0: std::unique_ptr<mongo::PlanExecutor, mongo::PlanExecutor::Deleter>::~unique_ptr()
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/ops/write_ops_exec.cpp:616:0: performSingleUpdateOp
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/ops/write_ops_exec.cpp:696:0: mongo::performUpdates(mongo::OperationContext*, mongo::write_ops::Update const&)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/commands/write_commands/write_commands.cpp:288:0: mongo::(anonymous namespace)::CmdUpdate::runImpl(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::BSONObjBuilder&)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/commands/write_commands/write_commands.cpp:223:0: mongo::(anonymous namespace)::WriteCommand::enhancedRun(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::BSONObjBuilder&)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/commands.cpp:356:0: mongo::Command::publicRun(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::BSONObjBuilder&)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/service_entry_point_mongod.cpp:508:0: runCommandImpl
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/service_entry_point_mongod.cpp:763:0: mongo::(anonymous namespace)::execCommandDatabase(mongo::OperationContext*, mongo::Command*, mongo::OpMsgRequest const&, mongo::rpc::ReplyBuilderInterface*) [clone .constprop.280]
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/service_entry_point_mongod.cpp:877:0: mongo::(anonymous namespace)::runCommands(mongo::OperationContext*, mongo::Message const&)::{lambda()#1}::operator()() const
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/service_entry_point_mongod.cpp:890:0: runCommands
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/service_entry_point_mongod.cpp:1144:0: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)
??:0:0: __libc_start_main
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_state_machine.cpp:370:0: mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_state_machine.cpp:431:0: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_state_machine.cpp:475:0: operator()
/opt/mongodbtoolchain/stow/gcc-v2.pOL/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
/opt/mongodbtoolchain/stow/gcc-v2.pOL/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_executor_synchronous.cpp:117:0: mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_state_machine.cpp:478:0: mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_state_machine.cpp:297:0: mongo::ServiceStateMachine::_sourceCallback(mongo::Status)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_state_machine.cpp:254:0: mongo::ServiceStateMachine::_sourceMessage(mongo::ServiceStateMachine::ThreadGuard)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_state_machine.cpp:428:0: mongo::ServiceStateMachine::_runNextInGuard(mongo::ServiceStateMachine::ThreadGuard)
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_state_machine.cpp:475:0: operator()
/opt/mongodbtoolchain/stow/gcc-v2.pOL/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::ServiceStateMachine::_scheduleNextWithGuard(mongo::ServiceStateMachine::ThreadGuard, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName, mongo::ServiceStateMachine::Ownership)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
/opt/mongodbtoolchain/stow/gcc-v2.pOL/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_executor_synchronous.cpp:134:0: operator()
/opt/mongodbtoolchain/stow/gcc-v2.pOL/include/c++/5.4.0/functional:1871:0: std::_Function_handler<void (), mongo::transport::ServiceExecutorSynchronous::schedule(std::function<void ()>, mongo::transport::ServiceExecutor::ScheduleFlags, mongo::transport::ServiceExecutorTaskName)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
/opt/mongodbtoolchain/stow/gcc-v2.pOL/include/c++/5.4.0/functional:2267:0: std::function<void ()>::operator()() const
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_entry_point_utils.cpp:57:0: mongo::(anonymous namespace)::runFunc(void*)
??:0:0: __libc_start_main
??:0:0: __libc_start_main
Suggested change in mongosymb.py output:
Unable to find source-code formatter for language: diff. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml
/data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/util/stacktrace_posix.cpp:175:0: mongo::printStackTrace(std::ostream&) /data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/util/signal_handlers_synchronous.cpp:182:0: mongo::(anonymous namespace)::printSignalAndBacktrace(int) /data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/util/signal_handlers_synchronous.cpp:278:0: mongo::(anonymous namespace)::abruptQuitWithAddrSignal(int, siginfo*, void*) - ??:0:0: __libc_start_main + libpthread.so.0 /data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/third_party/gperftools-2.5/src/linked_list.h:75:0: tcmalloc::SLL_PopRange(void**, int, void**, void**) ... /data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/db/service_entry_point_mongod.cpp:1144:0: mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&) - ??:0:0: __libc_start_main + imperva_mongo_lean3.so mongod_handle_request /data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_state_machine.cpp:370:0: mongo::ServiceStateMachine::_processMessage(mongo::ServiceStateMachine::ThreadGuard) ... /data/mci/0cfa6ba7e86ec6aa2439c24113c32d1f/src/src/mongo/transport/service_entry_point_utils.cpp:57:0: mongo::(anonymous namespace)::runFunc(void*) - ??:0:0: __libc_start_main - ??:0:0: __libc_start_main + libpthread.so.0 + libc.so.6 clone