Reduce noise in BSON perf-task benchmarks

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • None
    • None
    • Python Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Context

      The BSON encode/decode perf benchmarks are too noisy, making a small regression hard to distinguish. Network is not a factor for these tests, so we should be able to reduce the noise.

      While working on abi3 support for the driver and benchmarking on a perf host, I identified the following potential improvements.

      • Pin each benchmark to a physical core using os.sched_setaffinity (when available on the OS).
      • Use time.process_time_ns instead of time.monotonic.
      • Warm up before timing and discard the first ~20% of iterations.

      Definition of done

      • Validate and implement the suggested approach(es)

      Pitfalls

      • Run the benchmarks on 10-20 separate runs to compute the baseline in terms of Coefficient of Variant (CoV), computed as std dev / mean
      • For a given improvement, run the same number of times and compare the CoV.
      • Only keep a change only if it brings CoV to ≤1%, or reduces it by ~25% or more, otherwise it isn't worth the change.

            Assignee:
            Unassigned
            Reporter:
            Steve Silvester
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: