[SERVER-58946] Toolchain gcc-generated debug symbols may be malformed Created: 29/Jul/21  Updated: 08/Jan/23  Resolved: 27/Dec/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Robert Guo (Inactive) Assignee: Alex Neben
Resolution: Done Votes: 0
Labels: post-v4-toolchain
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
Assigned Teams:
Server Development Platform
Backwards Compatibility: Fully Compatible
Sprint: Dev Platform 2021-08-23, Dev Platform 2021-09-20, Dev Platform 2021-10-04, Dev Platform 2021-10-18, Dev Platform 2021-11-29
Participants:

 Description   

The toolchain gcc generated debug symbols can't be read entirely by the toolchain's llvm-symbolizer; there are line numbers missing and maybe other issues. This is possibly only affecting statically linked binaries as some shared library builds and possibly only mongod. Static mongos and shared libraries seem to be fine.

Using llvm-symbolizer-10 and above works. Drew tried 10 and I tried 12.

Copying some of Drew's research from Slack for context:

Running various versions of llvm-symbolizer

andrew@acm-workstation:~/asdf/dist-test/bin$ time llvm-symbolizer-7 --obj=./mongod <<< 0x2D5A0F0
mongo::LockRequestList::push_back(mongo::LockRequest*)
??:0:0
real	0m0.905s
user	0m0.828s
sys	0m0.061s
andrew@acm-workstation:~/asdf/dist-test/bin$ time llvm-symbolizer-8 --obj=./mongod <<< 0x2D5A0F0
mongo::LockRequestList::push_back(mongo::LockRequest*)
??:0:0
real	0m0.905s
user	0m0.816s
sys	0m0.073s
andrew@acm-workstation:~/asdf/dist-test/bin$ time llvm-symbolizer-9 --obj=./mongod <<< 0x2D5A0F0
mongo::LockRequestList::push_back(mongo::LockRequest*)
??:0:0
real	0m0.889s
user	0m0.810s
sys	0m0.060s
andrew@acm-workstation:~/asdf/dist-test/bin$ time llvm-symbolizer-10 --obj=./mongod <<< 0x2D5A0F0
mongo::LockRequestList::push_back(mongo::LockRequest*)
/data/mci/49914e6e89befead31fc1fecb83f2a54/src/src/mongo/db/concurrency/lock_request_list.h:74:25
real	0m7.524s
user	0m7.100s
sys	0m0.404s

Checking the debug symbols format with dwarfdump:

# mongod
# $ dwarfdump -ka ./mongod.debug
dwarfdump ERROR:  Failure reading CU header or DIE, corrupt DWARF:  DW_DLE_READ_ERROR(419). Read failed, corrupt object. Attempting to continue.
dwarfdump ERROR:  attempting to print .debug_info:  DW_DLE_READ_ERROR(419). Read failed, corrupt object (419)
attempting to continue.
dwarfdump ERROR:  dwarf_get_aranges:  DW_DLE_READ_ERROR(419). Read failed, corrupt object
 
# mongos
*** DWARF CHECK: DW_TAG_enumeration_type -> DW_AT_encoding: check the tag-attr combination ***
CU Name = src/mongo/s/mongos.cpp
CU Producer = GNU C++17 8.5.0 -march=sandybridge -mtune=generic -mprefer-vector-width=128 -ggdb -O2 -std=c++17 -fsized-deallocation -fasynchronous-unwind-tables -fno-omit-frame-pointer -fno-strict-aliasing -fstack-protector-strong -fno-builtin-memcmp -fPIE
DIE OFF = 0x00000036 GOFF = 0x00000036, Low PC = 0x00000000, High PC = unknown
DW_AT_encoding              DW_ATE_signed
*** DWARF CHECK: DW_TAG_enumeration_type -> DW_AT_encoding: check the tag-attr combination ***

readelf seems to work for both mongod and mongos with both showing debug symbols and source files.



 Comments   
Comment by Alex Neben [ 27/Dec/22 ]

Feel free to reopen if there is something I missed or am not understanding.

Comment by Alex Neben [ 27/Dec/22 ]

 

(v4-venv) ➜  mongo git:(master) ✗ /opt/mongodbtoolchain/v4/bin/dwarfdump -ka build/opt/mongo/db/mongod.debug*** DWARF CHECK: .debug_line: Address 0x0287e317 DW_LNE_end_sequence address does not exactly match high function addr:  0x0287e2e0 ***CU Name = (indexed string: 0x00000001)/run/user/1000/ccache-tmp/tmp.cpp_stdout.5kLIOJ.ii
CU Producer = (indexed string: 0x00000000)MongoDB clang version 12.0.1 (git@github.com:10gen/toolchain-builder.git ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce)
DIE OFF = 0x00000018 GOFF = 0x00000018, Low PC = 0x0287e2e0, High PC = 0x0287e2e9
Record = 14 Addr = 0x0287e317 [  57, 5] '/home/ubuntu/mongo/src/mongo/util/quick_exit.h'
ATE of 0 (0x0) is unknown to dwarfdump. Continuing.CU Name = (indexed string: 0x00000001)src/mongo/db/mongod_main.cpp
CU Producer = (indexed string: 0x00000000)MongoDB clang version 12.0.1 (git@github.com:10gen/toolchain-builder.git ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce)
DIE OFF = 0x00000050 GOFF = 0x0000322c, Low PC = 0x00000000, High PC = unknown
DW_AT_encoding              <Unknown ATE value 0x0>/opt/mongodbtoolchain/v4/bin/dwarfdump ERROR:  ERROR: dwarf_siblingof fails tracing siblings of a DIE.:  DW_DLE_DIE_ABBREV_LIST_NULL: There is no abbrev present for code 17617 in this compilation unit. The highest known code in any compilation unit is 17617 .. Attempting to continue.CU Name = (indexed string: 0x00000001)src/mongo/db/mongod_main.cpp
CU Producer = (indexed string: 0x00000000)MongoDB clang version 12.0.1 (git@github.com:10gen/toolchain-builder.git ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce)
DIE OFF = 0x00000050 GOFF = 0x0000322c, Low PC = 0x00000000, High PC = unknown/opt/mongodbtoolchain/v4/bin/dwarfdump ERROR:  printing .debug_info had a problem.:  DW_DLE_DIE_ABBREV_LIST_NULL: There is no abbrev present for code 17617 in this compilation unit. The highest known code in any compilation unit is 17617 .. Attempting to continue.CU Name = (indexed string: 0x00000001)src/mongo/db/mongod_main.cpp
CU Producer = (indexed string: 0x00000000)MongoDB clang version 12.0.1 (git@github.com:10gen/toolchain-builder.git ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce)
DIE OFF = 0x00000050 GOFF = 0x0000322c, Low PC = 0x00000000, High PC = unknownERROR: .eh_frame not loadable. dwarf_get_fde_list_eh DW_DLE_HEADER_LEN_BIGGER_THAN_SECSIZE:  The header length of 0x43434700 is larger than the section length of 0x602f84./opt/mongodbtoolchain/v4/bin/dwarfdump ERROR:  printing eh frame data had a problem.:  DW_DLE_HEADER_LEN_BIGGER_THAN_SECSIZE:  The header length of 0x43434700 is larger than the section length of 0x602f84.. Attempting to continue.1 Compilers detected:
01: (indexed string: 0x00000000)MongoDB clang version 12.0.1 (git@github.com:10gen/toolchain-builder.git ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce)1 Compilers verified:
01: errors =     2, (indexed string: 0x00000000)MongoDB clang version 12.0.1 (git@github.com:10gen/toolchain-builder.git ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce) 

 

 

Comment by Alex Neben [ 27/Dec/22 ]

/opt/mongodbtoolchain/v4/bin/llvm-symbolizer --obj=build/opt/mongo/db/mongod <<< 0x2D5A0F0
__gnu_cxx::__exchange_and_add_dispatch(int*, int)
/opt/mongodbtoolchain/revisions/ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce/stow/gcc-v4.8nU/lib/gcc/x86_64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/ext/atomicity.h:98:9
std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()
/opt/mongodbtoolchain/revisions/ecdb6f23a2cc2e822b8dc2eeb3f04c31f25c66ce/stow/gcc-v4.8nU/lib/gcc/x86_64-mongodb-linux/11.3.0/../../../../include/c++/11.3.0/bits/shared_ptr_base.h:180:10 

Comment by Alex Neben [ 27/Dec/22 ]

dwarfdump is still returning some errors in v4 but llvm-symbolizer seems to be working better with an optimized clang build. I am not sure what else to do as part of this ticket so I am going to close it.

Generated at Thu Feb 08 05:45:55 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.