-
Type: Task
-
Resolution: Fixed
-
Priority: Minor - P4
-
Affects Version/s: None
-
Component/s: None
-
Correctness
-
Fully Compatible
Full slack thread below, but the highlights for consideration as this one ticket or split out for several tickets:
- We should make the resmoke invocation blob display the TSAN_OPTIONS to configure
- It is not obvious that the "san" build profile includes address and undefined but not thread... Can we name them better? or at least add a 'tsan' option?
Slack Thread captured from #server-testing by charlie.swanson@mongodb.com
- charlie.swanson@mongodb.com: alex.neben@mongodb.com (or anyone familiar) should this list include "thread"? In order to reproduce TSAN BFs? <https://github.com/10gen/mongo/blob/0cc86e40e9fe1c279d650b59427f4ca96eb36be0/site_scons/mongo/build_profiles.py#L124>
- charlie.swanson@mongodb.com: cc will.buerger@mongodb.com
- charlie.swanson@mongodb.com: Oh wait, that doesn't seem to work for me:
```Checking if C compiler supports -fsanitize=undefined,address,thread... no
Failed to enable sanitizers with flag: -fsanitize=undefined,address,thread```
- william.qian@mongodb.com: Just a hunch, but can you check to see if `-ltsan` is included in the compiler flags?
- max.hirschhorn@mongodb.com: I didn't think ASan and TSan could be mixed together
- max.hirschhorn@mongodb.com: https://github.com/mongodb/mongo/blob/6d30cad3af7dcfdd963100d7d74548ef7ae464f4/etc/evergreen_yml_components/variants/sanitizer/test_dev.yml#L6[…]701
- charlie.swanson@mongodb.com: Yeah will.buerger@mongodb.com and I are giving it a shot now using this list: <https://github.com/10gen/mongo/blob/e17541934eb208c35475a158db4ef827b660eb21/etc/evergreen_yml_components/variants/sanitizer/test_dev.yml#L692>
- charlie.swanson@mongodb.com: jinx!
- charlie.swanson@mongodb.com: It is not obvious that "san" includes address and undefined but not thread... Can we name them better?
- william.qian@mongodb.com: They won't play well at all together, no. :upside_down_face:
- max.hirschhorn@mongodb.com: (might be worth adding a separate build profile for TSan to ninja/bazel)
- william.qian@mongodb.com: Or just never have concurrency bugs!
- charlie.swanson@mongodb.com: Relatedly, the instructions for reproducing a tsan failure say nothing about tsan:
```# This is a jstestfuzz suite and is dependent on the generated tests specific to this task execution.
- Having trouble reproducing your failure with this? Feel free to reach out in #server-testing.
mkdir -p jstestfuzz/ && \
rm -rf jstestfuzz/* && \
wget '<https://mciuploads.s3.amazonaws.com/mongodb-mongo-master/enterprise-rhel80-debug-tsan/939c396b2905b7c1da460e180ef5fb63bf5f0cbe/jstestfuzz/mongodb_mongo_master_enterprise_rhel80_debug_tsan_jstestfuzz_replication_3_linux_enterprise_939c396b2905b7c1da460e180ef5fb63bf5f0cbe_24_01_06_18_19_10-0.tgz>' -O jstests.tgz && \
tar -xf jstests.tgz -C jstestfuzz/ && \
rm jstests.tgz && \
buildscripts/resmoke.py run {}suites=jstestfuzz_replication -excludeWithAnyTags=tsan_incompatible --jobs=1 '-mongodSetParameters={logComponentVerbosity: {command: 2}}' --storageEngineCacheSizeGB=1```
- charlie.swanson@mongodb.com: Is that easily fixable? I feel like that comment about jstestfuzz is nice and there should be another one about compiling with special flags and then you also need to specify the 'san_options' when running resmoke I think?
- max.hirschhorn@mongodb.com: I think the Resmoke.py Invocation for Local Usage generally assumes you already have the appropriate binaries. It doesn't cover SCons/ninja invocations though I think generating similar instructions for compile_dist_test would. While engineers like to debug on their own machines spawning and setting up a host from the failing Evergreen task would download the already-built TSan binaries
- charlie.swanson@mongodb.com: Can we put that assumption in a comment then? :smile: Not how I do it
- charlie.swanson@mongodb.com: since then you can't really verify if you fixed it
- matt.kneiser@mongodb.com: It would be worth providing some more details on our hitchhiker guide: <https://wiki.corp.mongodb.com/pages/viewpage.action?pageId=80809667#ClangSanitizersandYou!-ThreadSanitizer(TSan)>
- matt.kneiser@mongodb.com: Max has some details on a personal page: <https://wiki.corp.mongodb.com/display/~max.hirschhorn/Customization+to+virtual+workstation>
- max.hirschhorn@mongodb.com: > and then you also need to specify the 'san_options' when running resmoke I think?
I agree we should make the resmoke invocation blob display the `TSAN_OPTIONS` to configure