[SERVER-72281] Provide Antithesis with tsan & asan enabled binaries Created: 20/Dec/22  Updated: 12/Dec/23  Resolved: 30/Nov/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Adrian Ryan Assignee: Trevor Guidry
Resolution: Duplicate Votes: 0
Labels: correctness-releaseability-working-group
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-83071 Provide antithesis with ASAN+UBSAN bi... Closed
Issue split
split to SERVER-83071 Provide antithesis with ASAN+UBSAN bi... Closed
Assigned Teams:
Correctness
Participants:

 Description   

In order to try to improve our testing of MongoDB's server, we want to look into using Mongo binaries with tsan and/or asan enabled. We'd first like to get versions of these binaries so that we can run some tests on our end, to understand whether these help us find issues and what performance implications they have inside our platform.

With these binaries we can run a controlled experiment & present a report on what we think the best use of testing is moving forward.



 Comments   
Comment by Ryan Berryhill [ 30/Nov/23 ]

This ticket is now subsumed by SERVER-83071

Comment by Judah Schvimer [ 09/Nov/23 ]

I split asan out to SERVER-83071.

Comment by Max Hirschhorn [ 03/Nov/23 ]

Yes and I would recommend starting out with a --sanitize=address binary first. And after we've gotten it running well in Antithesis infrastructure we can switch for a --sanitize=address,undefined binary next. I expect the --sanitize=address binary to be sufficient to ensure things like ASAN_OPTIONS environment variables and ulimit (also systemd?) virtual memory limits are configured appropriately. UBSan doesn't allocate shadow memory like ASan and TSan do and so it should be a much easier extension once we've done the first step.

Comment by Trevor Guidry [ 03/Nov/23 ]

My understanding is that ASAN and UBSAN can be combined but TSAN is standalone.

Comment by javi Arguello [ 02/Nov/23 ]

Awesome. Thanks trevor.guidry@mongodb.com . Let me know if there's anything you need from me in terms of pushing the images to our registry

Comment by javi Arguello [ 02/Nov/23 ]

Let's try as many as we can. Per Judah's message: "A/UB/TSAN (all the SANs we can)". Maybe we could start with whichever is easiest to send over to us? 

 

In terms of the tests, I believe we should try running the new binary with all of them multiple times. The more we can explore the search space, the better chance we have of finding a valuable bug 

Comment by Trevor Guidry [ 02/Nov/23 ]

I'm heads down focused on getting a project done by EOW but can take a look next week.

Comment by Steve Gross [ 02/Nov/23 ]

Excellent, good to know.

Which specific SAN variant(s)? do we want to try out, and for which test(s)?

Comment by javi Arguello [ 02/Nov/23 ]

#2: Antithesis will be adding tests temporarily to evaluate their utility and will credit all the test hours used for these new tests while the evaluation is ongoing  

Comment by Steve Gross [ 02/Nov/23 ]

Let's clarify:

  1. Are we simply changing existing tests to use SAN-enabled binaries, or are we adding tests that use SAN-enabled binaries?
  2. If we're adding tests, we should do so temporarily to evaluate their utility; Antithesis folks: please confirm that you'll credit us the test hours to try out additional tests in this case.
Comment by javi Arguello [ 02/Nov/23 ]

Bringing this card back to the forefront. After discussing with judah.schvimer@mongodb.com , we think it could be beneficial to run SAN enabled binaries inside Antithesis for exposing memory leaks important for the server architecture team. 

 

trevor.guidry@mongodb.com would this be something you could help us with? 

Comment by javi Arguello [ 14/Aug/23 ]

Jumping in here from the Antithesis side. In discussion with tausif.rahman@mongodb.com, this project was deemed lower priority and hence moved to the backlog

Generated at Thu Feb 08 06:21:20 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.