Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-69994

broken spawnhost setup for debugging C++ benchmark (bad globs?)

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 6.2.0-rc0
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • Fully Compatible
    • ALL
    • Execution Team 2022-10-17
    • 21

      I have a BF to debug.
      https://jira.mongodb.org/browse/BF-26477

      I have a core dump to load into gdb.
      https://spruce.mongodb.com/task/mongodb_mongo_master_rhel80_debug_asan_benchmarks_orphaned_7645dbbe88354f09ee8d84dc393a899cd067bc71_22_09_23_02_34_41/files?execution=0&sortBy=STATUS&sortDir=ASC

      The executable that failed is a C++ benchmark: service_executor_bm

      I asked Evergreen for a spawnhost to investigate this.
      Logging into the spawnhost, I don't get a useful environment.
      My ssh session is greeted with a lot of noise about failed glob patterns:

      Last login: Mon Sep 26 18:47:43 2022 from 207.251.78.34

Debuggable binaries:
ls: cannot access 'mongo*': No such file or directory
mongo core dumps:
ls: cannot access 'dump_mongo.*': No such file or directory
mongod core dumps:
ls: cannot access 'dump_mongod.*': No such file or directory
mongos core dumps:
ls: cannot access 'dump_mongos.*': No such file or directory
Core dumps from unknown processes (crashed processes typically found here):
-rw------- 1 ec2-user ec2-user 21211217920 Sep 23 08:08 dump_service_executo.112617.core

To examine a core dump, type 'gdb ./<binary> ./<core file>'

      I can see that a few symbolic links failed to be created.

      Looks like this was probably the outcome of some simple bash scripts that are not written to deal with unmatched globs. The * symlink is a dead symlink below, pointing to a bin/* file that doesn't exist

      [ec2-user@ip-10-128-152-165 debug]$ ls -al
total 20714080
lrwxrwxrwx 1 ec2-user ec2-user           5 Sep 26 18:42 '*' -> 'bin/*'
drwxrwxr-x 3 ec2-user ec2-user         130 Sep 26 18:42  .
drwxr-xr-x 5 ec2-user ec2-user          41 Sep 26 18:40  ..
lrwxrwxrwx 1 ec2-user ec2-user          53 Sep 26 18:40  .gdbinit -> /data/mci/source-mongodb-mongo-master-7645db/.gdbinit
drwxrwxr-x 6 ec2-user ec2-user         152 Sep 26 18:40  Boost-Pretty-Printer
lrwxrwxrwx 1 ec2-user ec2-user          57 Sep 26 18:40  buildscripts -> /data/mci/source-mongodb-mongo-master-7645db/buildscripts
-rw------- 1 ec2-user ec2-user 21211217920 Sep 23 08:08  dump_service_executo.112617.core
lrwxrwxrwx 1 ec2-user ec2-user          48 Sep 26 18:40  src -> /data/mci/source-mongodb-mongo-master-7645db/src

            Assignee:
            matt.kneiser@mongodb.com Matt Kneiser
            Reporter:
            billy.donahue@mongodb.com Billy Donahue
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: