[SERVER-48953] Add an option for resmoke to accept a replay file that lists tests. Created: 18/Jun/20 Updated: 29/Oct/23 Resolved: 20/Jul/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 4.7.0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Daniel Gottlieb (Inactive) | Assignee: | Daniel Gottlieb (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Backwards Compatibility: | Fully Compatible |
| Sprint: | STM 2020-06-29, STM 2020-07-13, STM 2020-07-27 |
| Participants: | |
| Story Points: | 2 |
| Description |
|
The patch now allows the following command line invocations:
and
If the contents of foo.txt are:
the invocation is analogous to:
Thus other command line flags such as --suite still take effect. Also noteworthy, resmoke will run the contents of the replay file in the order they are listed. Repeated test files will be run once per repetition; prior behavior was to dedup test files. Original ticket: The goal here is to also be compatible with using --jobs=N. The jobs flag will partition the test suite into different runners. When providing a seed, the each runner should deterministically run the same tests in the same order. |
| Comments |
| Comment by Githook User [ 20/Jul/20 ] | ||||||||||||
|
Author: {'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}Message: | ||||||||||||
| Comment by Daniel Gottlieb (Inactive) [ 02/Jul/20 ] | ||||||||||||
|
carl.worley noted that because jobs just pull the next job off a queue, a seed wouldn't give a lot of reproducibility. I researched what makes sense to do a bit last week with the help of a patch failure as inspiration. I think having test runs output a manifest file in the end does make sense. It seems tractible and would allow for other use cases (or maybe just simplify what can be done already on the command line) such as duplicating the same test file multiple times, or using an evergreen patch run's manifest as a starting point and pruning it to the tests one likes. The simple thing I want to accomplish with this ticket: Allow for a manifest file of the following format:
In this format there, any "job" (worker) can process a test file. There's no guarantee a single worker will process the same tests in repeated runs. This is effectively the same as listing out the files on the command line invocation. The bonus thing to accomplish:
The first column (number) is the job/worker that processed a test file. When consuming this manifest file, there's a guarantee that workers will work on the same tests in the same order. There's no guarantee that the actual tests being processed concurrently are the same between an evergreen patch/commit run and a local run. The worker values can be given in any order to ease how a run outputs its manifest file. E.g: A stable sort on the first field will result in an equivalent execution. Other goals:
|