-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Infrastructure, Performance Benchmarking
-
None
-
Python Drivers
-
None
-
None
-
None
-
None
-
None
-
None
Context
Our current performance tests experience significant variance due to inherent Python constraints such as garbage collection, event scheduling, and other environmental factors. We should explore options for reducing this variance. Some ideas:
Explicit garbage collection after each test iteration or benchmarkRunning every N (maybe 10?) test iterations in a separate Python processMore explicit warm-up iterations before measurement of each benchmarkAdd a proxy that introduces a fixed, constant latency to network operations for more consistent and real-world performance.
Rewrite the benchmarks to distribute the total load over processes rather than iterations within a process. Then also construct parsing logic for each result and take the average of the performance results as the final output.
Definition of done
Success will be indicated by a significant decrease in performance variability.
Pitfalls
Manual garbage collection and process management in Python can be complex and have unintended consequences.