[SERVER-74651] Fix implicit multiversion tests from not running in patches Created: 06/Mar/23 Updated: 27/Oct/23 Resolved: 28/Mar/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Jeffrey Zambory | Assignee: | Mikhail Shchatko |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Decision Automation Group
|
| Sprint: | DAG 2023-04-03 |
| Participants: | |
| Story Points: | 5 |
| Description |
|
Implicit multiversion tests are generated as part of the regular burn_in task generation for patches. These tasks are created to test if changing a test file will break it across a variety of different multiversion situations. These tasks do not exist as part of the original patch definition. They are generated as part of its lifecycle by version_burn_in_gen on a special variant called ! Enterprise RHEL 8.0 (implicit multiversion) that itself does not exist until after the generator has run. The only way to run these tasks currently is to wait until after the version_burn_in_gen task has run and then to reconfigure the patch to include the tasks on that new variant. This means it is very hard to run these tasks and we are lacking multiversion testing coverage in patches. These tests are valuable and we should be running them. A couple of ideas for fixing this
|
| Comments |
| Comment by Mikhail Shchatko [ 28/Mar/23 ] |
|
I've checked how burn-in works at the moment and can say that everything works as expected. Here is the patch with the dummy change in one of the js test: To give a more broad explanation the burn-in works in the following way. We have `version_burn_in_gen` task that generates all burn-in related tasks and variant even if no one chose to run them. The way evergreen works, as soon as those tasks/variants are generated, you will be able to see them in `Reconfigure tasks/variants` view in the patch. But it is not expected from users to do that to run those tasks. Instead of that we have `burn_in_tests_gen` and `burn_in_tags_gen` tasks on `! Shared Library Enterprise RHEL 8.0 (all feature flags)` and `! Enterprise Windows (all feature flags)` required variants. `burn_in_tests_gen` task is supposed to activate those generated tasks that appear on the same variant and `burn_in_tags_gen` is doing the same thing for tasks that are generated on a separated newly generated variants, which are `! Enterprise RHEL 8.0 (implicit multiversion)` and `! Enterprise RHEL 8.0 (inMemory)`. That means that `! Enterprise RHEL 8.0 (implicit multiversion)` variant with burn_in will run in case user selects to run a patch with all required variants (of course when the changed js test affects any implicit multiversion task, otherwise it will be no-op). |
| Comment by Alex Neben [ 06/Mar/23 ] |
|
I also wonder what we could add to catch this kind of thing from happening again? Maybe some data on the amount of tests run? I don't have a strong preference between solutions here. |