-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Performance Benchmarking
-
None
-
Python Drivers
-
None
-
None
-
None
-
None
-
None
-
None
Context
The performance suite measures data throughput: BSON encoding and decoding, bulk
CRUD, and GridFS. Nothing measures connection checkout, so changes to pool or
topology locking are invisible to CI.
PYTHON-5898 had to measure both of its proposed changes with throwaway
benchmarks. The pool change improved checkout throughput 6.6% at concurrency 20;
the proposed topology reader/writer lock made server selection ~30% slower and
was abandoned. Neither result would have been visible in CI.
Definition of done
- A benchmark measuring connection checkout throughput under concurrency,
following the existing PerformanceTest pattern in
test/performance/perf_test.py, with an async counterpart in
async_perf_test.py. - Reports operations_per_sec in the metrics list rather than reusing
megabytes_per_sec, which does not apply. - Covers both the uncontended path and the wait-queue path (concurrency above
maxPoolSize). - Runs in the existing performance-benchmarks variant, no new hardware, no
meaningful added wall-clock time. - Results appear in Evergreen's performance tooling and are comparable across
commits.
Pitfalls
- Driving checkout through ordinary operations measures the network and server,
not the driver. In PYTHON-5898, checkout ran ~190k ops/s in isolation while
real operations against localhost sustained ~8k ops/s. Exercise checkout
directly. - Do not instrument locks to report contention percentages. PYTHON-5898
measured topology lock contention at 0.016% of worker time at concurrency 20,
falling to 0.003% at 100 – not an actionable signal, and it requires patching
internals that later break. - Size the case so run-to-run variance is well below the effect sizes worth
catching. The two changes above were 6.6% and 30%. - Async and sync differ structurally: a single-threaded event loop versus threads
under the GIL. The same workload can behave differently; cover both. - Fix concurrency level and iteration count rather than deriving them from the
host, or results cannot be compared against history. - Anything comparing two builds must pin the interpreter version and confirm the
C extensions are present on both. An early PYTHON-5898 comparison was invalid
because the two sides ran different Python versions.
- is related to
-
PYTHON-5898 Reduce server selection and pool lock overhead on the checkout hot path
-
- In Progress
-