[SERVER-71211] gdb-index causing bugs in server gdb Created: 09/Nov/22  Updated: 29/Oct/23  Resolved: 17/Jan/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.3.0-rc0

Type: Bug Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Daniel Moody
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-50476 gdb.lookup_type on dynamically linked... Closed
Related
is related to SERVER-71098 Create additional gdb tests Open
Backwards Compatibility: Fully Compatible
Operating System: ALL
Participants:

 Description   

We need to remove gdb-index when generated with the gold linker in v3. This causes problems when debugging.



 Comments   
Comment by Daniel Moody [ 05/Jan/23 ]

max.hirschhorn@mongodb.com have you tried manually creating an index via gdb? I am working to integrate these commands into the build, they take a similar amount of time as when using the linker to generate the index, and far less memory. We would be able to use this to generate an index for static builds as well. here are some example commands (Note clang with dynamic link model falls back to DWARF4 format, the series of commands are different when using DWARF4):

/opt/mongodbtoolchain/v4/bin/gdb --batch --eval-command "save gdb-index -dwarf-5 build/install/bin" build/install/bin/mongod
/opt/mongodbtoolchain/v4/bin/objcopy --dump-section .debug_str=build/install/bin/mongod.debug_str.new build/install/bin/mongod
cat build/install/bin/mongod.debug_str >>build/install/bin/mongod.debug_str.new
/opt/mongodbtoolchain/v4/bin/objcopy --add-section .debug_names=build/install/bin/mongod.debug_names --set-section-flags .debug_names=readonly --update-section .debug_str=build/install/bin/mongod.debug_str.new build/install/bin/mongod build/install/bin/mongod 

Comment by Max Hirschhorn [ 05/Jan/23 ]

Thanks for the tip about CCFLAGS="-ggnu-pubnames" daniel.moody@mongodb.com. I tried out using v4 clang + lld + pubnames and am continuing to see the same error with absl::node_hash_map attempting to print a std::pair from a core dump.

$ git rev-parse HEAD
07f56293c96f4075eea9498aae088e3f186a3622
python ./buildscripts/scons.py --modules= --dbg=off --opt=on --link-model=dynamic --variables-files=./etc/scons/mongodbtoolchain_stable_clang.vars --variables-files=./etc/scons/experimental_unified_ninja.vars --ninja ICECC=icecc CCACHE=ccache CCFLAGS="-ggnu-pubnames"

(gdb) print *(mongo::ResourceCatalog*)0x559570d65450
$1 = {
  _mutex = { ... },
  _resources = absl::node_hash_map<mongo::ResourceId, absl::lts_20210324::flat_hash_set<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, mongo::StringMapHasher, mongo::StringMapEq, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> with 26 elems Traceback (most recent call last):
  File "buildscripts/gdb/mongo_printers.py", line 560, in children
    yield ('key', kvp['first'])
gdb.error: There is no member named first.
 
}

Comment by Daniel Moody [ 04/Jan/23 ]

Looks like the root cause may have been related to not using ggnu-pubnames option, and in the lld case you tested, the index may have been silently ignored?

 

This info came from https://jira.mongodb.org/browse/SERVER-50476?focusedCommentId=5090631&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-5090631

Comment by Max Hirschhorn [ 04/Jan/23 ]

Thanks daniel.moody@mongodb.com, I followed the steps in this comment on SERVER-71098 and don't see the gdb.error anymore when printing out the ResourceCatalog. It could be the switch to lld and that way of building the gdb-index addressed this. I could also imagine the gdb-index issue only happens some of the time based on the linking order for the mongod libraries. I don't have a good way to tell given my limited understanding of why symbol resolution failed initially.

$ git rev-parse HEAD
07f56293c96f4075eea9498aae088e3f186a3622

(gdb) print *(mongo::ResourceCatalog*)0x556aa633b450
$1 = {
  _mutex = { ... },
  _resources = absl::node_hash_map<mongo::ResourceId, absl::lts_20210324::flat_hash_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, mongo::StringMapHasher, mongo::StringMapEq, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >> with 26 elems  = {
    [{
      _fullHash = 7390088490467259311
    }] = absl::flat_hash_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >> with 1 elems  = {"local.oplog.rs"},
    [{
      _fullHash = 8816103937297484293
    }] = absl::flat_hash_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >> with 1 elems  = {"local.system.replset"},

Comment by Daniel Moody [ 03/Jan/23 ]

max.hirschhorn@mongodb.com recently we switched to lld linker as the default  (SERVER-72324), do you think this issue is fixed by the new linker? Its gdb index seems to work better. I saw the repro in the comment, but I didn't try it yet, do you think you could give it a repro and see if thinks are working better?

Comment by Daniel Moody [ 28/Dec/22 ]

I have a working branch where the build will invoke gdb to create and save the index in the binary. Also we will be be switching to lld linker very soon, and it may generate valid gdb-index, I need to test it a bit after we switch to lld linker.

Comment by Max Hirschhorn [ 28/Dec/22 ]

See also this comment on SERVER-71098 for an example where absl::node_hash_map fails to resolve correctly when printing out all of the decorations on the global ServiceContext.

Comment by Alex Neben [ 09/Nov/22 ]

https://mongodb.slack.com/archives/C04143WUDC1/p1668005734780559

Generated at Thu Feb 08 06:18:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.