[DRIVERS-2649] Ensure unified spec tests use the lowest possible spec version Created: 12/Jun/23 Updated: 21/Sep/23 |
|
| Status: | Backlog |
| Project: | Drivers |
| Component/s: | Unified Test Runner |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Unknown |
| Reporter: | Andreas Braun | Assignee: | Unassigned |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Driver Changes: | Not Needed | ||||||||
| Description |
SummaryThe unified test runner specification requires tests to use the lowest possible spec version that supports all features that are used. This is not enforced however, and adhering to this requirement is difficult for spec authors. A script to check files could help authors ensure their tests comply with the requirement, and allows us to also check it during the CI process. The test process is as follows (for each test file):
MotivationWho is the affected end user?Driver authors. How does this affect the end user?Using a higher than necessary spec version can cause problems for drivers that haven't yet implemented the changes required for that version. For example, version 1.13 introduced changes for logging, so any driver that hasn't yet implemented changes for the logging spec will not be able to run such files. Only tests that use logging functionality (or any other functionality added later) should specify a 1.13 requirement to ensure maximum compatibility with drivers. How likely is it that this problem or use case will occur?This has occurred multiple times and has generally been difficult to enforce (both as author and as reviewer). If the problem does occur, what are the consequences and how severe are they?Drivers can't run spec tests even when the test runner supports all features in the tests. Is this issue urgent?Not urgent Is this ticket required by a downstream team?No Is this ticket only for tests?No functional impact and only test changes Acceptance Criteria
|
| Comments |
| Comment by Githook User [ 13/Jun/23 ] |
|
Author: {'name': 'Jérôme Tamarelle', 'email': 'jerome@tamarelle.net', 'username': 'GromNaN'}Message: CI Script to be created later as part of DRIVERS-2649 |