[SERVER-72613] Speed up taking core dumps with the hang analyzer Created: 07/Jan/23  Updated: 29/Oct/23  Resolved: 09/Jan/23

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 5.0.15, 6.0.4, 6.3.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Max Hirschhorn
Resolution: Fixed Votes: 0
Labels: tig-hanganalyzer
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
related to SERVER-75070 increase hang analyzer self test time... Closed
is related to SERVER-70802 Mongod data directory and FTDC files ... Closed
is related to SERVER-56167 Guarantee hang analyzer collects core... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.2, v6.0, v5.0
Sprint: Sharding NYC 2023-01-09, Sharding NYC 2023-01-23
Participants:

 Description   

The hang analyzer uses the gcore command (aka generate-core-file) within a GDB session to generate a core dump. By default, GDB will load the debug symbols for the process when the attach command is run. The gcore script uses the --readnever option when invoking gdb to skip the expense of loading debug symbols before taking the core dump. This cost is amortized in the hang analyzer because it'll use a single GDB session for all processes of the same type (i.e. a single gdb process attaches to all mongod processes). However we can use --readnever in the hang analyzer when it does the first round of attach commands to only take core dumps and avoid the cost altogether.

# `</dev/null' to avoid touching interactive terminal if it is
# available but not accessible as GDB would get stopped on SIGTTIN.
"$binary_path/@GDB_TRANSFORM_NAME@" </dev/null \
    --nx --batch --readnever \
    -ex "set pagination off" -ex "set height 0" -ex "set width 0" \
    "${dump_all_cmds[@]}" \
    -ex "attach $pid" -ex "gcore $prefix.$pid" -ex detach -ex quit

https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=8a6a85134d78531c6adb72a888844f7cfa3a5c56



 Comments   
Comment by Githook User [ 09/Jan/23 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-72613 Run gdb with --readnever when only taking core dumps.

(cherry picked from commit afd5fda2f81419f45f321d425bbcf9127c738ac8)
Branch: v6.0
https://github.com/mongodb/mongo/commit/0a997440edbb4bb546068acf2396ba5d254f3df0

Comment by Githook User [ 09/Jan/23 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-72613 Run gdb with --readnever when only taking core dumps.

(cherry picked from commit afd5fda2f81419f45f321d425bbcf9127c738ac8)
Branch: v5.0
https://github.com/mongodb/mongo/commit/076efd1f8c546e72e18d9522d6bdfaec4e7a4811

Comment by Githook User [ 09/Jan/23 ]

Author:

{'name': 'Max Hirschhorn', 'email': 'max.hirschhorn@mongodb.com', 'username': 'visemet'}

Message: SERVER-72613 Run gdb with --readnever when only taking core dumps.
Branch: master
https://github.com/mongodb/mongo/commit/afd5fda2f81419f45f321d425bbcf9127c738ac8

Generated at Thu Feb 08 06:22:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.