[Spike] Reduce performance test Python environment noise

XMLWordPrintableJSON

    • Python Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Context

      Our current performance tests experience significant variance due to inherent Python constraints such as garbage collection, event scheduling, and other environmental factors. We should explore options for reducing this variance. Some ideas:

      1. Explicit garbage collection after each test iteration or benchmark
      2. Running every N (maybe 10?) test iterations in a separate Python process
      3. More explicit warm-up iterations before measurement of each benchmark
      4. Add a proxy that introduces a fixed, constant latency to network operations for more consistent and real-world performance.

      Rewrite the benchmarks to distribute the total load over processes rather than iterations within a process. Then also construct parsing logic for each result and take the average of the performance results as the final output.

      Definition of done

      Success will be indicated by a significant decrease in performance variability.

      Pitfalls

      Manual garbage collection and process management in Python can be complex and have unintended consequences.

            Assignee:
            Unassigned
            Reporter:
            Noah Stapp
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: