-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Performance, Performance Benchmarking
-
None
-
None
-
Python Drivers
-
None
-
None
-
None
-
None
-
None
-
None
Context
The perf tests themselves are expensive to run and are not run by default on PRs.
We also do not have a good way to do A/B testing when making bson changes locally.
When we do compare two builds, running them as separate back-to-back passes is noisy: the perf host drifts between passes, so a real change is hard to distinguish from host drift. Interleave the two builds within one run to cancel that drift.
Definition of done
- Add a separate "just benchmark-bson" hook that takes an optional "--compare". The default compare target is the base commit, but you can choose another ref.
- The compare run interleaves the candidate and the base ref (alternate A/B within one session) and reports the ratio of the interleaved medians, rather than measuring each build in a separate pass.
- Add an Evergreen task that runs on PRs, compares against the base ref, and fails if the candidate falls below a threshold. Still run on the perf hosts, but the run time will be much shorter. Run it with the pr tag.
Pitfalls
- Make sure the test is reliable: run several alternating A/B cycles and take the median (or min-of-medians) of the interleaved pairs, and only fail if the regression is consistent.
- Interleaving is what cancels host drift; a non-interleaved comparison can report a false regression. Measured on a perf host, a non-interleaved run produced a false ~9% deep_bson.decode regression that disappeared once the two builds were interleaved.
- blocks
-
PYTHON-6097 Improve BSON handling of string keys and values
-
- Needs Triage
-
-
PYTHON-6098 Improve bson handling of datetime
-
- Needs Triage
-
-
PYTHON-6099 Improve bson tz-aware datetime encoding performance
-
- Needs Triage
-
-
PYTHON-6107 Improve BSON encode dispatch with exact-type fast path
-
- Needs Triage
-
-
PYTHON-6108 Fix is_dict_class detection to use the fast decode path
-
- Needs Triage
-
- is blocked by
-
PYTHON-6103 Reduce noise in BSON perf-task benchmarks
-
- Needs Triage
-