[SERVER-67131] Test HWAsan Created: 08/Jun/22  Updated: 29/Oct/23  Resolved: 14/Sep/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Alex Neben Assignee: Alex Neben
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: Dev Platform 2022-06-27, Dev Platform 2022-07-11, Dev Platform 2022-07-25
Participants:

 Comments   
Comment by Alex Neben [ 14/Sep/22 ]

Here was the evergreen

- <<: *rhel80-debug-asan-all-feature-flags-template
  name: rhel80-debug-hwasan-ubsan-all-feature-flags
  display_name: "~ Shared Library HWASAN and UBSAN Enterprise RHEL 8.0 DEBUG (all feature flags)"
  run_on:
  - amazon2-arm64-small
  expansions:
    compile_flags: --variables-files=etc/scons/mongodbtoolchain_${toolchain_version}_clang.vars --dbg=on --opt=on --allocator=system --sanitize=undefined,hwaddress --ssl MONGO_DISTMOD=amazon2 --ocsp-stapling=off -j$(grep -c ^processor /proc/cpuinfo) --link-model=dynamic
    crypt_task_compile_flags: SHLINKFLAGS_EXTRA="-Wl,-Bsymbolic -Wl,--no-gnu-unique"
      CCFLAGS="-fno-gnu-unique"
    san_options: >-
      UBSAN_OPTIONS="print_stacktrace=1:external_symbolizer_path=/opt/mongodbtoolchain/v3/bin/llvm-symbolizer"
      LSAN_OPTIONS="suppressions=etc/lsan.suppressions:report_objects=1:external_symbolizer_path=/opt/mongodbtoolchain/${toolchain_version}/bin/llvm-symbolizer"
      HWASAN_OPTIONS="detect_leaks=1:check_initialization_order=true:strict_init_order=true:abort_on_error=1:disable_coredump=0:handle_abort=1:strict_string_checks=true:detect_invalid_pointer_pairs=1:external_symbolizer_path=/opt/mongodbtoolchain/${toolchain_version}/bin/llvm-symbolizer"
    large_distro_name: amazon2-arm64-large
    multiversion_architecture: arm64
    multiversion_architecture_42_or_later: aarch64
    multiversion_edition: enterprise
    multiversion_platform: amazon2
    multiversion_skip_install_for_40: 'true'
    scons_cache_scope: none
    shared_scons_pruning: false
    test_flags: --excludeWithAnyTags=incompatible_with_amazon_linux,requires_ldap_pool,uses_pykmip,requires_v4_0
    use_scons_cache: false 

Comment by Alex Neben [ 14/Sep/22 ]

Using our compiled clang there is a hang and error when compiling with hwasan. I think I have hit the timebox limit on this ticket and am going to merge this without enabling new tests.

scons: *** [build/optdebug/mongo/util/net/ssl_manager_openssl.dyn.o] Error -9 

Here is a sample evergreen run
https://spruce.mongodb.com/version/63211aa99ccd4e77c0d08dbd/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&statuses=failed-umbrella,failed,task-timed-out,test-timed-out,known-issue&variant=%5Erhel80-debug-hwasan-ubsan-all-feature-flags%24

Comment by Alex Neben [ 14/Sep/22 ]

I think this means that the vendored clang by amazon linux 2022 is missing that library and we could get around that by compiling our own clang. I am trying again with clang in the v4 toolchain.

Comment by Alex Neben [ 14/Sep/22 ]

Tried with a newer clang (12.0.1) and had the following error

clang -o build/scons/optdebug/sconf_temp/conftest_df286a1d2f67e69d030b4eff75ca7e12_0.o -c -std=c11 -Werror -fasynchronous-unwind-tables -ggdb -Wall -Wsign-compare -Wno-unknown-pragmas -Winvalid-pch -fno-omit-frame-pointer -fno-strict-aliasing -O2 -march=armv8.2-a -mtune=generic -Wno-unused-local-typedefs -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-range-compare -Wno-tautological-constant-compare -Wno-tautological-unsigned-zero-compare -Wno-tautological-unsigned-enum-zero-compare -Wno-unused-const-variable -Wno-missing-braces -Wno-inconsistent-missing-override -Wno-potentially-evaluated-expression -Wno-unused-lambda-capture -fstack-protector-strong -fsanitize=undefined,hwaddress -fno-omit-frame-pointer -fno-sanitize-recover -fno-sanitize=vptr -fPIE -DPCRE2_STATIC -DMONGO_USE_VISIBILITY -D_XOPEN_SOURCE=700 -D_GNU_SOURCE -DADDRESS_SANITIZER -DUNDEFINED_BEHAVIOR_SANITIZER build/scons/optdebug/sconf_temp/conftest_df286a1d2f67e69d030b4eff75ca7e12_0.c
clang -o build/scons/optdebug/sconf_temp/conftest_df286a1d2f67e69d030b4eff75ca7e12_0_ad960b59c6e27002afacc72e9a4010bd -Wl,--fatal-warnings -Wl,--no-as-needed -pthread -Wl,-z,now -fuse-ld=gold -fstack-protector-strong -fsanitize=undefined,hwaddress -fno-sanitize=vptr -pie -rdynamic build/scons/optdebug/sconf_temp/conftest_df286a1d2f67e69d030b4eff75ca7e12_0.o -lm -lresolv -llzma
/bin/ld.gold: error: cannot open /usr/lib64/clang/12.0.1/lib/linux/libclang_rt.hwasan-aarch64.a: No such file or directory
build/scons/optdebug/sconf_temp/conftest_df286a1d2f67e69d030b4eff75ca7e12_0.o:conftest_df286a1d2f67e69d030b4eff75ca7e12_0.c:function hwasan.module_ctor: error: undefined reference to '__hwasan_init'
clang-12: error: linker command failed with exit code 1 (use -v to see invocation)
scons: Configure: no 

Comment by Alex Neben [ 14/Sep/22 ]

Update ran with toolchain v3 on 5.15 and had no luck

Comment by Alex Neben [ 14/Sep/22 ]

Looks like it has to be v5.4 maybe based on some pretty sketchy guess work
https://github.com/torvalds/linux/commit/6cf5354c1c4b74fd2e5527db084f163e9d4dae4e
https://source.android.com/docs/security/test/tagged-pointers

 

Comment by Alex Neben [ 14/Sep/22 ]

Looks like it might be fixed in a newer version of the kernel
https://www.kernel.org/doc/html/latest/arm64/tagged-address-abi.html

Comment by Alex Neben [ 14/Sep/22 ]

Results
https://www.kernel.org/doc/Documentation/arm64/tagged-pointers.txt
Doesn't seem to work

[js_test:import_collection_command_sanity] JSTest src/mongo/db/modules/enterprise/jstests/live_import/import_collection_command_sanity.js started with pid 3759.
[js_test:import_collection_command_sanity] HWAddressSanitizer:DEADLYSIGNAL
[js_test:import_collection_command_sanity] ==3759==ERROR: HWAddressSanitizer: UNKNOWN SIGNAL on unknown address 0x03e800000eaf (pc 0xffff841084b0 bp 0xffffc20c4a10 sp 0xffffc20c4a10 T3759)
[js_test:import_collection_command_sanity]     #0 0xffff841084ac in gsignal (/lib64/libpthread.so.0+0x134ac)
[js_test:import_collection_command_sanity]     #1 0xffff859e8edc in mongo::breakpoint() /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/util/debugger.cpp:72:5
[js_test:import_collection_command_sanity]     #2 0xffff8596f648 in mongo::fassertFailedWithLocation(int, char const*, unsigned int) /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/util/assert_util.cpp:220:5
[js_test:import_collection_command_sanity]     #3 0xffff85968864 in mongo::(anonymous namespace)::Source::refill(unsigned char*, unsigned long) /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/platform/random.cpp:153:21
[js_test:import_collection_command_sanity]     #4 0xffff85968864 in mongo::SecureUrbg::State::get() /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/platform/random.cpp:204
[js_test:import_collection_command_sanity]     #5 0xffff858c3900 in long std::uniform_int_distribution<long>::operator()<mongo::SecureUrbg>(mongo::SecureUrbg&, std::uniform_int_distribution<long>::param_type const&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.RcJ/lib/gcc/aarch64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/uniform_int_dist.h:275:21
[js_test:import_collection_command_sanity]     #6 0xffff858c1a34 in long std::uniform_int_distribution<long>::operator()<mongo::SecureUrbg>(mongo::SecureUrbg&) /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.RcJ/lib/gcc/aarch64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/uniform_int_dist.h:166:24
[js_test:import_collection_command_sanity]     #7 0xffff858c1a34 in long mongo::RandomBase<mongo::SecureUrbg>::_nextAny<long>() /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/platform/random.h:160
[js_test:import_collection_command_sanity]     #8 0xffff858c1a34 in mongo::RandomBase<mongo::SecureUrbg>::nextInt64() /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/platform/random.h:121
[js_test:import_collection_command_sanity]     #9 0xffff858c1a34 in mongo::_mongoInitializerFunction_OIDGeneration(mongo::InitializerContext*) /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/bson/oid.cpp:64
[js_test:import_collection_command_sanity]     #10 0xffff85823578 in std::function<void (mongo::InitializerContext*)>::operator()(mongo::InitializerContext*) const /opt/mongodbtoolchain/revisions/c6da1cf7f0b4b60d53566305e59857d3d540dcf7/stow/gcc-v3.RcJ/lib/gcc/aarch64-mongodb-linux/8.5.0/../../../../include/c++/8.5.0/bits/std_function.h:687:14
[js_test:import_collection_command_sanity]     #11 0xffff85821598 in mongo::Initializer::executeInitializers(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/base/initializer.cpp:159:9
[js_test:import_collection_command_sanity]     #12 0xffff85822800 in mongo::runGlobalInitializers(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/base/initializer.cpp:205:32
[js_test:import_collection_command_sanity]     #13 0xffff85822a6c in mongo::runGlobalInitializersOrDie(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/base/initializer.cpp:222:25
[js_test:import_collection_command_sanity]     #14 0xffff85d72b8c in mongo::mongo_main(int, char**) /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/shell/mongo_main.cpp:717:9
[js_test:import_collection_command_sanity]     #15 0xaaaabd030da4 in main /data/mci/af102a9d3794a783370361236af4c455/src/src/mongo/shell/mongo.cpp:42:22
[js_test:import_collection_command_sanity]     #16 0xffff83f5dd60 in __libc_start_main (/lib64/libc.so.6+0x1fd60)
[js_test:import_collection_command_sanity]     #17 0xaaaabcfcce20 in _start (/data/mci/3c0b7f07a89ff8b7ef2c1eb6dfc9bb3e/src/dist-test/bin/mongo+0x22e20)
[js_test:import_collection_command_sanity] 
[js_test:import_collection_command_sanity] HWAddressSanitizer can not provide additional info.
[js_test:import_collection_command_sanity] SUMMARY: HWAddressSanitizer: UNKNOWN SIGNAL (/lib64/libpthread.so.0+0x134ac) in gsignal
[js_test:import_collection_command_sanity] ==3759==ABORTING
[js_test:import_collection_command_sanity] Looking for stacktrace files in '/data/db/job0'
[js_test:import_collection_command_sanity] No failure logs/stacktrace files found, skipping symbolization 

Generated at Thu Feb 08 06:07:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.