Core analysis task generation fails when gdb errors trying to resolve the binary of a coredump

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Minor - P4
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • DevProd Test Infrastructure
    • DevProd Test Infra 2026-08-25
    • 0
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      [2026/07/25 23:21:04.129] Running command 'subprocess.exec' ('gen hang analyzer tasks script') in function 'generate hang analyzer tasks' (step 30.3 of 42) in block 'post'.
      [2026/07/25 23:21:04.140] Found python 3.13 in v5 toolchain, setting python to /opt/mongodbtoolchain/v5/bin/python3.13
      [2026/07/25 23:21:04.351] ['', '/opt/mongodbtoolchain/v5/lib/python313.zip', '/opt/mongodbtoolchain/v5/lib/python3.13', '/opt/mongodbtoolchain/v5/lib/python3.13/lib-dynload', '/data/mci/88a907d16c365a664f122562e7f8ea28/venv/lib/python3.13/site-packages']
      [2026/07/25 23:21:04.379] ['', '/data/mci/88a907d16c365a664f122562e7f8ea28/src', '/opt/mongodbtoolchain/v5/lib/python313.zip', '/opt/mongodbtoolchain/v5/lib/python3.13', '/opt/mongodbtoolchain/v5/lib/python3.13/lib-dynload', '/data/mci/88a907d16c365a664f122562e7f8ea28/venv/lib/python3.13/site-packages']
      [2026/07/25 23:21:04.386] python set to /data/mci/88a907d16c365a664f122562e7f8ea28/venv/bin/python and python version: Python 3.13.11
      [2026/07/25 23:21:04.386] python buildscripts/resmokelib/hang_analyzer/gen_hang_analyzer_tasks.py
      [2026/07/25 23:24:14.682] cd src
      [2026/07/25 23:24:14.682] activate_venv
      [2026/07/25 23:24:14.682] echo $python $@
      [2026/07/25 23:24:14.682] $python "$@"
      [2026/07/25 23:24:14.682] Traceback (most recent call last):
      [2026/07/25 23:24:14.682]   File "/data/mci/88a907d16c365a664f122562e7f8ea28/src/buildscripts/resmokelib/hang_analyzer/gen_hang_analyzer_tasks.py", line 437, in <module>
      [2026/07/25 23:24:14.682]     task_config = generator.generate()
      [2026/07/25 23:24:14.683]   File "/data/mci/88a907d16c365a664f122562e7f8ea28/src/buildscripts/resmokelib/hang_analyzer/gen_hang_analyzer_tasks.py", line 132, in generate
      [2026/07/25 23:24:14.683]     cores = self.find_cores()
      [2026/07/25 23:24:14.683]   File "/data/mci/88a907d16c365a664f122562e7f8ea28/src/buildscripts/resmokelib/hang_analyzer/gen_hang_analyzer_tasks.py", line 245, in find_cores
      [2026/07/25 23:24:14.683]     binary_name, bin_version = dumpers.dbg.get_binary_from_core_dump(core_file)
      [2026/07/25 23:24:14.683]                                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
      [2026/07/25 23:24:14.683]   File "/data/mci/88a907d16c365a664f122562e7f8ea28/src/buildscripts/resmokelib/hang_analyzer/dumper.py", line 977, in get_binary_from_core_dump
      [2026/07/25 23:24:14.683]     process = subprocess.run(
      [2026/07/25 23:24:14.683]         [dbg, "-batch", "--quiet", "-ex", f"core {core_file_path}"],
      [2026/07/25 23:24:14.683]     ...<2 lines>...
      [2026/07/25 23:24:14.683]         text=True,
      [2026/07/25 23:24:14.683]     )
      [2026/07/25 23:24:14.683]   File "/opt/mongodbtoolchain/v5/lib/python3.13/subprocess.py", line 577, in run
      [2026/07/25 23:24:14.683]     raise CalledProcessError(retcode, process.args,
      [2026/07/25 23:24:14.683]                              output=stdout, stderr=stderr)
      [2026/07/25 23:24:14.683] subprocess.CalledProcessError: Command '['/opt/mongodbtoolchain/v5/bin/gdb', '-batch', '--quiet', '-ex', 'core  dump_eviction-ser 1.187693.core']' returned non-zero exit status 1.
      [2026/07/25 23:24:15.216] Command 'subprocess.exec' ('gen hang analyzer tasks script') in function 'generate hang analyzer tasks' (step 30.3 of 42) in block 'post' failed: process encountered problem: exit code 1.
      [2026/07/25 23:24:15.216] Finished command 'subprocess.exec' ('gen hang analyzer tasks script') in function 'generate hang analyzer tasks' (step 30.3 of 42) in block 'post' in 3m11.08675082s.
      

      https://parsley.corp.mongodb.com/evergreen/mongodb_mongo_master_enterprise_rhel8_debug_tsan_roll_back_incremental_feature_flags_unshard_collection_fuzzer_4_linux_enterprise_e12cee8684849ae2aa421d89af0f7edca119999d_26_07_25_05_06_08/0/task?bookmarks=0%2C5585&selectedLineRange=L5238-L5267&shareLine=5238

      We should make this give more output from what went wrong with GDB and fix the root cause.

            Assignee:
            Sean Lyons
            Reporter:
            Sean Lyons
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: