[SERVER-41063] Improve stacktrace handling of binary names Created: 09/May/19 Updated: 27/May/22 Resolved: 27/May/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Diagnostics, Internal Code |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Kevin Pulo | Assignee: | Robert Guo (Inactive) |
| Resolution: | Done | Votes: | 1 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Participants: | |||||
| Case: | (copied to CRM) | ||||
| Description |
|
Currently, stacktrace output from mongosymb.py doesn't include the paths/filenames of any binaries. This is a problem because there are times when this information is relevant and important, but is missing from the main symbolised stacktrace that is being used diagnostically. Instead there is typically a line with no value like one of:
This is because:
In addition, the stacktrace printing code does not output Dl_info::dli_fname (the filename of the binary for that frame) in the JSON backtrace (even though it outputs the basename of this in the "classic" output), opting instead to output dl_phdr_info::dlpi_name in the somap. The Linux man page for dl_iterate_phdr(3) says:
However, this is not always true — for example, LD_PRELOAD causes dlpi_name to be null/empty. In these cases, dli_fname still contains the correct filename for the frame. In the example below see the mongod_handle_request frame, which is actually in imperva_mongo_lean3.so, but this library is not mentioned (by name) in the somap. There is also some weird relocation thing going on (ASR?), because the addr bases for symbols in libs like libc and libpthread don't match the bases reported in the somap. This further compounds the symbol resolution problems above, and is another reason to include dli_fname for each frame. In the example below, the libc and libpthread frames report addresses starting with 0x7fa6a..., whereas in the somap they have base addresses starting with 0x7f6b7...:
The specific changes I'd like to suggest are:
Example:
Suggested change in mongosymb.py output:
|
| Comments |
| Comment by Billy Donahue [ 04/Apr/22 ] | |||
|
There are some requests for mongosymb.py here, so I've added this to PM-2295 "Improved Mongo Symbolizer" for further investigation there. As for the C++ side of this, which is the missing filenames, I believe this may already be handled at least partially. As part of loading a symbol's metadata, we have this extra mergeDlInfo stage now.
Leaving the ticket open for the mongosymb.py request though. | |||
| Comment by Andrew Morrow (Inactive) [ 24/Jul/19 ] | |||
|
Moving out of the libunwind epic and into quick wins. This is really about improvements to mongosymb.py, and not directly related to integrating libunwind. | |||
| Comment by Andy Schwerin [ 09/May/19 ] | |||
|
These are worthwhile improvements to consider, but should be broken out into more tickets. For example, I think the frame base address vs soname base address discrepancy is either a bug or an unhandled side effect of LD_PRELOADand should be treated as a bug. Maybe the rest could be left as a single improvement in this ticket. |