[SERVER-69991] C++ benchmarks running under ASAN? Created: 26/Sep/22  Updated: 05/Dec/22  Resolved: 30/Sep/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Billy Donahue Assignee: Backlog - Performance Team
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-65359 Don't run benchmarks on DEBUG/*SAN bu... Closed
Related
Assigned Teams:
Product Performance
Participants:
Linked BF Score: 21

 Description   

We've been assigned a Buildfailure related to a runtime problem caused by a C++ benchmark running under DEBUG and ASAN.

https://jira.mongodb.org/browse/BF-26477

While I haven't yet investigated the cause here, it seems to be some kind of resource exhaustion inside the benchmark framework.

The thing about it is that I'm not sure of the value of running benchmarks under ASAN at all, since they're all about measuring performance, and ASAN imposes enormous performance penalties that would make the results of these computationally intensive tests useless.

Should ASAN and DEBUG builds skip C++ benchmarks?



 Comments   
Comment by Billy Donahue [ 26/Sep/22 ]

Note: Linked BF-26477 was a false positive induced entirely by ASAN's interference with thread creation, during a benchmark that spawns threads in a loop.

Comment by Alex Neben [ 26/Sep/22 ]

We can certainly exclude ASAN and  DEBUG builds from running benchmarks but I would argue that we shouldn't. While I do agree that the perf numbers we get here are meaningless ASAN and DEBUG both run benchmarks with more coverage meaning that we can uncover testing bugs by doing so. I think that is worth ensuring that these stay green rather than just excluding them.

Generated at Thu Feb 08 06:14:59 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.