[SERVER-80303] Investigate bug in which gdb usage is not possible on a spawnhost if the failure came from a *san variant Created: 22/Aug/23 Updated: 06/Nov/23 Resolved: 26/Oct/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.2.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Trevor Guidry | Assignee: | Daniel Moody |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Correctness
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Operating System: | ALL | ||||||||
| Participants: | |||||||||
| Description |
|
If an engineer attempts to debug a variant failure on a spawnhost, AND the variant was a *san configuration, gdb will NOT work on the spawnhost.
On sanitizer variants we build with separate-debug=off which includes the debugsymbols inside of the binaries. The setup_spawnhosts scripts does some stuff which modifies the binary which makes the binary not work with gdb. If you download the binary from evergreen it does work.
Max h hypothesized that adding the .gdb_index section into the executable itself (separate-debug=off) causes gdb to not recognize the executable as having produced the core dump and therefore refuses to load the debug symbols from the executable
Original slack thread: https://mongodb.slack.com/archives/CR8SNBY0N/p1692653657122299 |
| Comments |
| Comment by Githook User [ 26/Oct/23 ] |
|
Author: {'name': 'Daniel Moody', 'email': 'dmoody256@gmail.com', 'username': 'dmoody256'}Message: |
| Comment by Max Hirschhorn [ 24/Aug/23 ] |
|
I did an experiment to see if it would be possible to use objcopy and strip to split the debug info into a separate file (such that, if successful, then we could build the index on the separate debug files like we do for other artifacts). GDB did not recognize the executable file after it was stripped as being the one which produced the core dump which precluded the ability to load the debug symbols. I think the only choice may be to skip generating a gdb index when the executable was compiled with --separate-debug=off. That is, the setup_spawnhost_coredump script will leave the executable files entirely alone in this case. GDB can then interpret the core dump with its startup time being slower due to the absence of a gdb index. |
| Comment by Trevor Guidry [ 24/Aug/23 ] |
|
The problem is that on spawnhosts gdb does not correctly work with the binaries that are outputted from out setup spawnhost script if they were built with `separate-debug=off`. Currently this is limited to our sanitizer variants because those are the ones we build with `separate-debug=off`. |