Summary

The unified test runner specification requires tests to use the lowest possible spec version that supports all features that are used. This is not enforced however, and adhering to this requirement is difficult for spec authors. A script to check files could help authors ensure their tests comply with the requirement, and allows us to also check it during the CI process.

The test process is as follows (for each test file):

• Read spec version from test file
• Validate test file against indicated schema version
• Validate against previous schema version, expect failure
• If previous task does not fail, keep repeating with older versions until validation failure is achieved
• Once the test fails validation against a schema version X.Y, X.Y+1 is the lowest required schema version and can be printed in the output.

Motivation

Who is the affected end user?

Driver authors.

How does this affect the end user?

Using a higher than necessary spec version can cause problems for drivers that haven't yet implemented the changes required for that version. For example, version 1.13 introduced changes for logging, so any driver that hasn't yet implemented changes for the logging spec will not be able to run such files. Only tests that use logging functionality (or any other functionality added later) should specify a 1.13 requirement to ensure maximum compatibility with drivers.

How likely is it that this problem or use case will occur?

This has occurred multiple times and has generally been difficult to enforce (both as author and as reviewer).

If the problem does occur, what are the consequences and how severe are they?

Drivers can't run spec tests even when the test runner supports all features in the tests.

Is this issue urgent?

Not urgent

Is this ticket required by a downstream team?

No

Is this ticket only for tests?

No functional impact and only test changes

Acceptance Criteria

• Create a script that verifies that each test file uses the lowest possible schema version
• Set up CI workflow to run script and fail CI if a file could use a lower spec version.
• Ideally, the CI step would create an error in the affected file, which is then shown in the "Code" tab of a pull request. This can be done using Workflow Commands