[SERVER-49857] ASAN Ubuntu 18.04 build variant did not symbolize its output Created: 24/Jul/20 Updated: 29/Oct/23 Resolved: 24/Aug/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 4.7.0, 4.4.2 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Max Hirschhorn | Assignee: | Ryan Egesdahl (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Backport Requested: |
v4.4
|
||||||||||||||||
| Sprint: | Dev Platform 2020-08-10, Dev Platform 2020-08-24, Dev Platform 2020-09-07 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Linked BF Score: | 42 | ||||||||||||||||
| Description |
|
Despite the /opt/mongodbtoolchain/v3/bin/ directory being added to the PATH environment variable for the started the mongos process, ASan did not symbolize its output. Could /opt/mongodbtoolchain/v3/bin/llvm-symbolizer actually not be present? https://clang.llvm.org/docs/AddressSanitizer.html#symbolizing-the-reports
|
| Comments |
| Comment by Githook User [ 14/Oct/20 ] |
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: The previous in SAN build, which caused dependency cycles and missing symbols. We |
| Comment by Githook User [ 13/Oct/20 ] |
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: SAN The toolchain llvm-symbolizer was never actually in PATH despite the (cherry picked from commit 20ed5d51cb1c82597f65967be69b81e6e72c0413) |
| Comment by Ryan Egesdahl (Inactive) [ 24/Aug/20 ] |
|
Symbolizing is actually happening but the lack of debuginfo is causing line numbers and source code not to be shown. We've just made discovering the correct llvm-symbolizer more robust here. A request to extract debuginfo during sanitizer builds has been opened as |
| Comment by Githook User [ 22/Aug/20 ] |
|
Author: {'name': 'Ryan Egesdahl', 'email': 'ryan.egesdahl@mongodb.com', 'username': 'deriamis'}Message: SAN The toolchain llvm-symbolizer was never actually in PATH despite the |
| Comment by Andrew Morrow (Inactive) [ 21/Aug/20 ] |
|
max.hirschhorn - I believe symbolization is happening, but it appears that the debug info files are not present so the symbolizer can't provide line numbers. robert.guo - I thought we were always pulling down the debug info at the beginning of each task? For instance this task https://evergreen.mongodb.com/task/mongodb_mongo_master_ubuntu1804_debug_asan_disk_wiredtiger_patch_e8b0acc393d181a50301a10ae436dc580b335858_5f3586d03066156e86056d18_20_08_13_18_30_59##comparehashes=e8b0acc393d181a50301a10ae436dc580b335858&threads=all downloads the debug info here: https://evergreen.mongodb.com/task_log_raw/mongodb_mongo_master_ubuntu1804_debug_asan_disk_wiredtiger_patch_e8b0acc393d181a50301a10ae436dc580b335858_5f3586d03066156e86056d18_20_08_13_18_30_59/0?type=T#L17 If we just unpacked it for sanitizer builds, I think line numbers would happen again. |
| Comment by Ryan Egesdahl (Inactive) [ 31/Jul/20 ] |
|
It looks like what's happening here is that the variant_path_suffix we set on the build variant is not being added to PATH during execute_resmoke_tests for some reason. I can confirm that the symbolizer is used as intended when ASAN_SYMBOLIZER_PATH is set in the build variant: I question the reason why we started adding *_SYMBOLIZER_PATH to the sanitizer build variants. It should be working if it were actually being added to the path, as I have confirmed locally. Something seems to be wrong with how we're using variant_build_path here, or with how it ends up functioning during resmoke execution. I'm following up that issue now to determine what the path forward here should be. |
| Comment by Ryan Egesdahl (Inactive) [ 30/Jul/20 ] |
|
I think this might be related to not having ASAN_SYMBOLIZER_PATH in etc/evergreen.yml the way we do in SConstruct, and maybe we need to do that because we can't fully trust PATH to be right. I'm not entirely convinced of that, but it is a difference that might matter. I don't really know how to reproduce this and verify that fact yet, though. I'm working on that. |