MFP non-emergency and emergency release process requires perf runs as a release quality gate to ensure that newer plugin does not regress in performance. When a new plugin is released, Release quality can give the release operator go/no-go based on its performance cost that is repeatable and written down.

Reference: Each release is compared against the plugin version deployed today. For the first rollout that is the no-op plugin; after that, the currently deployed plugin. The release playbook linked above already specifies this.

We had planned to use a test-only "parse-all" plugin as a fixed ceiling instead. That is dropped. It was never measured, and it is not a real ceiling: parse-all parses and validates every field but decides nothing, so a filter that also matches on values can cost more than parse-all does.

Deliverables -

• How to produce a comparison?
  • The same tasks on both sides, so the results are comparable
  • Same server compile on both sides so that plugin must be only variable
• How to read it to determine regression - pass/fail/marginal?
  • How to take measurements? which metrics to measure e.g. p50, p99 of read latency, throughput etc.
  • If there is a marginal increase, then should we consider a re-run?
• A short verdict after analysis about the result that goes back to the Release operator, preferably recorded in a release ticket.
  • For emergency release - Perf gate is skipped (run is deferred) but the results are backfilled soon after.
• Nature of engagement - Who on Release team should run perf checks and how to raise a request? A ticket? Urgency