[SERVER-74651] Fix implicit multiversion tests from not running in patches Created: 06/Mar/23  Updated: 27/Oct/23  Resolved: 28/Mar/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Jeffrey Zambory Assignee: Mikhail Shchatko
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Decision Automation Group
Sprint: DAG 2023-04-03
Participants:
Story Points: 5

 Description   

Implicit multiversion tests are generated as part of the regular burn_in task generation for patches. These tasks are created to test if changing a test file will break it across a variety of different multiversion situations.

These tasks do not exist as part of the original patch definition. They are generated as part of its lifecycle by version_burn_in_gen on a special variant called ! Enterprise RHEL 8.0 (implicit multiversion) that itself does not exist until after the generator has run. The only way to run these tasks currently is to wait until after the version_burn_in_gen task has run and then to reconfigure the patch to include the tasks on that new variant. 

This means it is very hard to run these tasks and we are lacking multiversion testing coverage in patches. These tests are valuable and we should be running them. 

A couple of ideas for fixing this

  1. Special case multiversion burn_in tests and have them activate themselves automatically on every patch
  2. Move these tasks to live on a different variant that is present at patch creation time (or make ! Enterprise RHEL 8.0 (implicit multiversion) available at creation time). This way people can schedule these tasks much easier. 


 Comments   
Comment by Mikhail Shchatko [ 28/Mar/23 ]

I've checked how burn-in works at the moment and can say that everything works as expected.

Here is the patch with the dummy change in one of the js test:
https://spruce.mongodb.com/version/6422dbc42a60ed9895a4b234/tasks?page=0&sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC&taskName=burn_in

To give a more broad explanation the burn-in works in the following way. We have `version_burn_in_gen` task that generates all burn-in related tasks and variant even if no one chose to run them. The way evergreen works, as soon as those tasks/variants are generated, you will be able to see them in `Reconfigure tasks/variants` view in the patch. But it is not expected from users to do that to run those tasks. Instead of that we have `burn_in_tests_gen` and `burn_in_tags_gen` tasks on `! Shared Library Enterprise RHEL 8.0 (all feature flags)` and `! Enterprise Windows (all feature flags)` required variants. `burn_in_tests_gen` task is supposed to activate those generated tasks that appear on the same variant and `burn_in_tags_gen` is doing the same thing for tasks that are generated on a separated newly generated variants, which are `! Enterprise RHEL 8.0 (implicit multiversion)` and `! Enterprise RHEL 8.0 (inMemory)`. That means that `! Enterprise RHEL 8.0 (implicit multiversion)` variant with burn_in will run in case user selects to run a patch with all required variants (of course when the changed js test affects any implicit multiversion task, otherwise it will be no-op).

Comment by Alex Neben [ 06/Mar/23 ]

I also wonder what we could add to catch this kind of thing from happening again? Maybe some data on the amount of tests run? I don't have a strong preference between solutions here.

Generated at Thu Feb 08 06:28:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.