[DRIVERS-2466] Test invalid spec tests in check_schema_version.sh Created: 10/Oct/22  Updated: 06/Jan/23

Status: Backlog
Project: Drivers
Component/s: Unified Test Runner
Fix Version/s: None

Type: Task Priority: Minor - P4
Reporter: Jeremy Mikola Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
Driver Changes: Not Needed

 Description   

Summary

In this PR comment from DRIVERS-2385, I noticed that the schema version in the unified test format's Makefile was outdated. The original purpose of that Makefile was the validate unified tests across all specs using the latest schema version and also test the spec's "invalid" tests, which should fail validation.

DRIVERS-1767 introduced a CI workflow to validate spec tests against their exact "schemaVersion", but notably omits the "invalid" tests. I propose that we remove this Makefile and update check_schema_version.sh to test "invalid" spec tests.

Additionally, we can look into optimizing the CI script to execute ajv once per distinct schema file. It currently invokes ajv once per test file, which is much slower than the older Makefile.

Motivation

Who is the affected end user?

Spec editors.

How does this affect the end user?

Schema files may not be fully tested.

Is this issue urgent?

No.

Is this ticket required by a downstream team?

No.

Is this ticket only for tests?

Yes.


Generated at Thu Feb 08 08:25:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.