Currently the query optimizer creates a set of candidate plans and then ranks them. Based on the existing tests we only validate the assumption of a specific winning plan. The testing gap we have is the true ranked winning plan vs. the actual costed (via CBR). Then additionally the quality of our cost model strongly depends on how good we can sample and how good the cost model is calibrated.

To create a data driven analysis on the quality of the ranker and plan exploration this ticket should:

Create plan alternatives without any restrictions (wrt. enumeration limitations)
Create plan alternatives with the restrictions in place (aka release configuration)

Create a ranking with CBR based on the cost and the sample for the above 2 plan space.
Create a ranking by executing the query against the database (cold cache) and then rank the plan alternatives base on the end to end latency.

A comparison of the 4 rankings will provide a quality metric of the enumerator as well as the rankers calibration and cost model. To create trackable metrics we should:

• Compare the first N elements of the ranking which should match between the synthetically created ranking and the "true" ranking (each for full exploration and limited exploration).
• Use an Kendall’s Tau to determine the quality of the synthetic ranking respectively how to