-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: BSON, Performance Benchmarking
-
None
-
None
-
Python Drivers
-
None
-
None
-
None
-
None
-
None
-
None
Context
The BSON encode/decode perf benchmarks are too noisy, making a small regression hard to distinguish. Network is not a factor for these tests, so we should be able to reduce the noise.
While working on abi3 support for the driver and benchmarking on a perf host, I identified the following potential improvements.
- Pin each benchmark to a physical core using os.sched_setaffinity (when available on the OS).
- Use time.process_time_ns instead of time.monotonic.
- Warm up before timing and discard the first ~20% of iterations.
Definition of done
- Validate and implement the suggested approach(es)
Pitfalls
- Run the benchmarks on 10-20 separate runs to compute the baseline in terms of Coefficient of Variant (CoV), computed as std dev / mean
- For a given improvement, run the same number of times and compare the CoV.
- Only keep a change only if it brings CoV to ≤1%, or reduces it by ~25% or more, otherwise it isn't worth the change.
- blocks
-
PYTHON-6096 Add BSON benchmarking regression test support
-
- Needs Triage
-