The query-correctness-tests cover behavior which can be influenced by feature flags. This means we cannot expect the same test to behave consistently across vanilla and all-feature-flags variants. In the short term, the query-correctness-test will only be run on all-feature-flags variants. The alternative is to exclude running on the all-feature-flags variants. The rationale is that all-feature-flags variants are testing changes we are introducing and should be tested eagerly. If we excluded all-feature-flags variants, we would need to update tests when feature flags are enabled. The potential risk is that some bug may be introduced which is hidden by a feature flag. Ultimately, we should handle this in a more granular way by allowing either different expected results or allowing tests to be disabled on some configurations/variants.