[SERVER-80303] Investigate bug in which gdb usage is not possible on a spawnhost if the failure came from a *san variant Created: 22/Aug/23  Updated: 06/Nov/23  Resolved: 26/Oct/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.2.0-rc0

Type: Bug Priority: Major - P3
Reporter: Trevor Guidry Assignee: Daniel Moody
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-74299 spawnhost script should not create gd... Closed
Assigned Teams:
Correctness
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

If an engineer attempts to debug a variant failure on a spawnhost, AND the variant was a *san configuration, gdb will NOT work on the spawnhost.

 

On sanitizer variants we build with separate-debug=off which includes the debugsymbols inside of the binaries. The setup_spawnhosts scripts does some stuff which modifies the binary which makes the binary not work with gdb. If you download the binary from evergreen it does work.

 

Max h hypothesized that adding the .gdb_index section into the executable itself (separate-debug=off) causes gdb to not recognize the executable as having produced the core dump and therefore refuses to load the debug symbols from the executable

 

Original slack thread: https://mongodb.slack.com/archives/CR8SNBY0N/p1692653657122299



 Comments   
Comment by Githook User [ 26/Oct/23 ]

Author:

{'name': 'Daniel Moody', 'email': 'dmoody256@gmail.com', 'username': 'dmoody256'}

Message: SERVER-80303 make spawnhost script avoid gdbindex for embedded debug files
Branch: master
https://github.com/mongodb/mongo/commit/9e9571ac1b1aedccbbdfad5c8776d71577d074ff

Comment by Max Hirschhorn [ 24/Aug/23 ]

I did an experiment to see if it would be possible to use objcopy and strip to split the debug info into a separate file (such that, if successful, then we could build the index on the separate debug files like we do for other artifacts). GDB did not recognize the executable file after it was stripped as being the one which produced the core dump which precluded the ability to load the debug symbols.

I think the only choice may be to skip generating a gdb index when the executable was compiled with --separate-debug=off. That is, the setup_spawnhost_coredump script will leave the executable files entirely alone in this case. GDB can then interpret the core dump with its startup time being slower due to the absence of a gdb index.

Comment by Trevor Guidry [ 24/Aug/23 ]

The problem is that on spawnhosts gdb does not correctly work with the binaries that are outputted from out setup spawnhost script if they were built with `separate-debug=off`. Currently this is limited to our sanitizer variants because those are the ones we build with `separate-debug=off`.

Generated at Thu Feb 08 06:43:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.