[SERVER-48730] Loading core dump on spawn host with gdb isn't reading debug symbols from shared objects automatically Created: 11/Jun/20  Updated: 14/Jul/20  Resolved: 14/Jul/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Andrew Morrow (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-46296 Migrate non-windows ! builders to --l... Closed
Operating System: ALL
Sprint: Dev Platform 2020-06-29, Dev Platform 2020-07-13, Dev Platform 2020-07-27
Participants:
Linked BF Score: 9

 Description   

My typical workflow is to extract tarballs for the executables, debug symbols, and core dumps into the source-mongodb-mongo-master-xx directory and then to invoke gdb as follows. For --link-model=dynamic builds, gdb fails to read the debug symbols from the dist-test/lib/ directory.

$ /opt/mongodbtoolchain/gdb/bin/gdb ./dist-test/bin/mongod ./dump_js.15583.core
...
warning: Could not load shared library symbols for 605 libraries, e.g. /data/mci/fb4e7f494e41a1298b97bc36d5110734/src/dist-test/bin/../lib/libmongod_main.so.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
...
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
                                        No          /data/mci/fb4e7f494e41a1298b97bc36d5110734/src/dist-test/bin/../lib/libmongod_main.so

Creating the /data/mci/fb4e7f494e41a1298b97bc36d5110734/src/ directory and extracting the executables and debug symbols (i.e. the dist-test/ directory) into it addresses this issue but is a hassle.

Setting the following search path for shared objects also addresses the issue for me. Is there a way to make this more automatic?

(gdb) set solib-search-path ./dist-test/lib
Reading symbols from /data/mci/source-mongodb-mongo-master-601e8c/dist-test/lib/libmongod_main.so...
Reading symbols from /data/mci/source-mongodb-mongo-master-601e8c/dist-test/lib/libmongod_main.so.debug...
...



 Comments   
Comment by Max Hirschhorn [ 11/Jun/20 ]

resmoke.py spawns processes using their absolute path. (The subprocess module simulates the execvpe() behavior.) Drew believes that the absolute path to the shared objects is being encoded in the core dump even though the executable only stores a relative path. Note that the /data/mci/fb4e7f494e41a1298b97bc36d5110734 directory in question is the one from when the test failure occurred and not from the compile task.

[2020/06/09 08:37:08.216] Making new folder for task execution: /data/mci/fb4e7f494e41a1298b97bc36d5110734
[2020/06/09 08:37:08.216] Reporting task started.
[2020/06/09 08:37:08.307] cleaning up processes for task: mongodb_mongo_master_enterprise_rhel_62_64_bit_dynamic_parallel_compatibility_601e8c7fccf22d676de89518ef129253edaca127_20_06_09_07_55_40

https://evergreen.mongodb.com/task_log_raw/mongodb_mongo_master_enterprise_rhel_62_64_bit_dynamic_parallel_compatibility_601e8c7fccf22d676de89518ef129253edaca127_20_06_09_07_55_40/0?type=E#L2

lldb similarly shows the shared objects from the core dump as having an absolute path.

$ /opt/mongodbtoolchain/v3/bin/lldb ./dist-test/bin/mongod -c ./dump_js.15583.core
...
(lldb) image list
[  0] A3557CEC-91DF-1A5D-08CD-68B5BC37974A-4A1007CB                    /data/mci/source-mongodb-mongo-master-601e8c/dist-test/bin/mongod
      /data/mci/source-mongodb-mongo-master-601e8c/dist-test/bin/mongod.debug
[  1] 08F634A1-D22D-EFF0-0461-D50A7699DACD-C97657BF 0x00007fffd8fff000 [vdso] (0x00007fffd8fff000)
[  2] E18B6376 0x00007fd58497a000 /data/mci/fb4e7f494e41a1298b97bc36d5110734/src/dist-test/lib/libmongod_main.so (0x00007fd58497a000)

Generated at Thu Feb 08 05:17:56 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.