[SERVER-69994] broken spawnhost setup for debugging C++ benchmark (bad globs?) Created: 26/Sep/22 Updated: 29/Oct/23 Resolved: 10/Oct/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 6.2.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Billy Donahue | Assignee: | Matt Kneiser |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Sprint: | Execution Team 2022-10-17 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Linked BF Score: | 21 | ||||||||||||||||
| Description |
|
I have a BF to debug. I have a core dump to load into gdb. The executable that failed is a C++ benchmark: service_executor_bm I asked Evergreen for a spawnhost to investigate this.
I can see that a few symbolic links failed to be created. Looks like this was probably the outcome of some simple bash scripts that are not written to deal with unmatched globs. The * symlink is a dead symlink below, pointing to a bin/* file that doesn't exist
|
| Comments |
| Comment by Githook User [ 10/Oct/22 ] | |||
|
Author: {'name': 'Matt Kneiser', 'email': 'matt.kneiser@mongodb.com', 'username': 'themattman'}Message: | |||
| Comment by Matt Kneiser [ 04/Oct/22 ] | |||
|
I just realized that the messages in this ticket's description refer to the .profile script that is added as a welcome message. I will patch that up. | |||
| Comment by Max Hirschhorn [ 27/Sep/22 ] | |||
Thanks billy.donahue@mongodb.com, I hadn't realized we were always uploading the C++ benchmark binaries to S3. It sounds like the setup_spawnhost_coredump script is missing handling for extracting the tarball.
| |||
| Comment by Billy Donahue [ 27/Sep/22 ] | |||
|
The globbing problems can be handled with shopt -s nullopt or shopt -s failglob in the script, depending on whether failed expansion should be an error or just a null string. The script's glob behavior is one thing. The s3 uploads of benchmark executables (the thing that exposed the scripting bug) is another thing. SDP could probably route this or handle it best. Assigning to them. | |||
| Comment by Billy Donahue [ 27/Sep/22 ] | |||
|
Well yes. I want a better debugging experience, but I'm also worried about the uncontrolled symlink creation when globs fail. If a glob fails and the literal * is echoed by the glob, we're going to be echoing weird message to the screen and running dangerous commands with wildcard arguments. That feels like an accident waiting to happen, with behavior dependent on what other files are laying around to fill the wildcard. I was able to recover the benchmark executables from the compile task and scp them into the spawnhost and get my gdb rolling, but it sure took a lot of figuring out. | |||
| Comment by Billy Donahue [ 26/Sep/22 ] | |||
|
These tarballs in the data directory are empty.
(This command is showing that thes 61-byte tarballs are valid tarballs containing no files) |