Add BSON benchmarking regression test support

XMLWordPrintableJSON

    • None
    • Python Drivers
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?
    • None
    • None
    • None
    • None
    • None
    • None

      Context

      The perf tests themselves are expensive to run and are not run by default on PRs.
      We also do not have a good way to do A/B testing when making bson changes locally.
      When we do compare two builds, running them as separate back-to-back passes is noisy: the perf host drifts between passes, so a real change is hard to distinguish from host drift. Interleave the two builds within one run to cancel that drift.

      Definition of done

      • Add a separate "just benchmark-bson" hook that takes an optional "--compare". The default compare target is the base commit, but you can choose another ref.
      • The compare run interleaves the candidate and the base ref (alternate A/B within one session) and reports the ratio of the interleaved medians, rather than measuring each build in a separate pass.
      • Add an Evergreen task that runs on PRs, compares against the base ref, and fails if the candidate falls below a threshold. Still run on the perf hosts, but the run time will be much shorter. Run it with the pr tag.

      Pitfalls

      • Make sure the test is reliable: run several alternating A/B cycles and take the median (or min-of-medians) of the interleaved pairs, and only fail if the regression is consistent.
      • Interleaving is what cancels host drift; a non-interleaved comparison can report a false regression. Measured on a perf host, a non-interleaved run produced a false ~9% deep_bson.decode regression that disappeared once the two builds were interleaved.

            Assignee:
            Unassigned
            Reporter:
            Steve Silvester
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: