[SERVER-61149] Enable one-to-many mapping between generated tasks and resmoke.py suites Created: 30/Oct/21 Updated: 23/Oct/23 |
|
| Status: | Open |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Robert Guo (Inactive) | Assignee: | [DO NOT ASSIGN] Backlog - DevProd Correctness |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Correctness
|
| Participants: |
| Description |
|
There is currently a one-to-one mapping of generated tasks and resmoke suites, as indicated by the single GeneratedSuite object. As we optimize the workflow to be based on a number of factors to optimize for coverage and latency, the notion of a "test suite" that consists of many "JS test"s will no longer be applicable. Instead, a "task" should be a logical grouping of validation steps for a portion of a certain feature, which is similar to the definition of most suites today. Note that there is additional higher-level groupings to capture the complete set of steps needed to validate a certain feature, module, or product, through existing mechanisms like Evergreen task tags, taskgroups, or build variants. E.g. multiversion tasks currently adds the version information to the suite to enable running with different combinations. This makes sense at the suite level if someone wants to reproduce a failure for a certain test, but is not necessary for Evergreen, because we don't have and don't expect to have a way of deciding if only one of the versions should be run for 99% of use cases. A similar notion can be added for groupings of suites that are considered "regression tests" for test suites not containing tests that are actively being worked on. The technical change involves changing the task generation logic into a pipeline instead of its current distinctive two step process of splitting the suite based on task history, and generating the tasks. The pipeline would consist of an arbitrary number of stages for arbitrary task/suite types. suite split can be one of the stages that acts on logical grouping of tests, instead of globally. For the multiversion example, we can split based on if the last version is last-continuous or last-its since there may be performance differences between the two versions. Note that the suite split strategies do not have to change since we are still splitting based on task history. Most the changes there is in the caller and restructuring the task generation orchestrator. |