Context

Right now, we compare the results of the benchmarks run in the "perf" task with the "stable region" from a set of previous waterfall builds, and post the diff of those times on PRs. That has proved to be quite noisy and isn't providing any value right now. We need to find a way to reduce the noise in the PR benchmark output so that only real performance diffs are called out.

At least one source of noise is that individual benchmark runs often differ a lot from the baseline, but that difference doesn't stay consistent through multiple runs. We really need to collect data from multiple runs (5 or more) in the PR benchmark task to make sure the results are valid.

It's not clear if continuing to use the "perfcomp" tool is the right approach, or if we should use a completely different approach for PR benchmarks.

Definition of done

• Use "benchstat" instead of "perfcomp" to report PR benchmark diffs.
  • Requires running the benchmarks for the base commit and PR commit multiple times to collect enough samples for "benchstat" to report statistically significant changes.
  • Build a tool to take the "benchstat" output and convert it into a Github comment.

Pitfalls

What should the implementer watch out for? What are the risks?