When our macOS patch builds fail, we have very limited information available (for an example, check the comments). It's basically limited to function names, with no line number available.
It gets worse after downloading the artifacts and trying to load the core dump into GDB - there is no debug info or source available, so looking at a core dump from Evergreen is basically pointless.
I did a little research on this (and it's probably wrong), but my understanding is that the source location is distributed around various .o files. For our build, it's a hardcoded /data/mci path, which isn't going to exist if you're loading the core file into GDB after an Evergreen test failure.
On a Linux spawn host it would be possible to do one of the following:
- objdump -S to "manually" find the source location
- Make a symlink with the /data/mci name
Unfortunately macOS doesn't have an equivalent for the first option, and even if it did, the symbols are distributed around a pile of .o files instead of living in one place. The second option (a symlink) isn't possible on the macOS shared hosts due to permissions issues.
For what it's worth, the .o files definitely have debug info - running dwarfdump on them generates something sane-looking.
It's not clear to me how the server does this, from a slack thread it seems there's a way to do this for the server - that could be worth investigating.
Next steps:
- Investigate how the server teams includes debug info in their macOS builds
- Investigate other macOS debugging options
- Check if the server's approach can be applied to WiredTiger
Definition of done: either a ticket with clear next steps, or fixed with working source information in Evergreen failures.