[SERVER-48953] Add an option for resmoke to accept a replay file that lists tests. Created: 18/Jun/20  Updated: 29/Oct/23  Resolved: 20/Jul/20

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.7.0

Type: Improvement Priority: Major - P3
Reporter: Daniel Gottlieb (Inactive) Assignee: Daniel Gottlieb (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: STM 2020-06-29, STM 2020-07-13, STM 2020-07-27
Participants:
Story Points: 2

 Description   

The patch now allows the following command line invocations:

resmoke run --replayFile foo.txt

and

resmoke run @foo.txt

If the contents of foo.txt are:

jstests/concurrency/fsm_workloads/indexed_insert_multikey.js
jstests/concurrency/fsm_workloads/indexed_insert_2dsphere.js

the invocation is analogous to:

resmoke run jstests/concurrency/fsm_workloads/indexed_insert_multikey.js jstests/concurrency/fsm_workloads/indexed_insert_2dsphere.js

Thus other command line flags such as --suite still take effect.

Also noteworthy, resmoke will run the contents of the replay file in the order they are listed. Repeated test files will be run once per repetition; prior behavior was to dedup test files.

Original ticket:
When running a set of tests with resmoke, one has the option of running them in alphabetical order or totally random. It would be nice if the random runs could at least use a deterministic order when provided a seed.

The goal here is to also be compatible with using --jobs=N. The jobs flag will partition the test suite into different runners. When providing a seed, the each runner should deterministically run the same tests in the same order.



 Comments   
Comment by Githook User [ 20/Jul/20 ]

Author:

{'name': 'Daniel Gottlieb', 'email': 'daniel.gottlieb@mongodb.com', 'username': 'dgottlieb'}

Message: SERVER-48953: Allow resmoke test files to be listed in a replay file.
Branch: master
https://github.com/mongodb/mongo/commit/c0be456afed815cfc490930a89085233292e3b3f

Comment by Daniel Gottlieb (Inactive) [ 02/Jul/20 ]

carl.worley noted that because jobs just pull the next job off a queue, a seed wouldn't give a lot of reproducibility. I researched what makes sense to do a bit last week with the help of a patch failure as inspiration. I think having test runs output a manifest file in the end does make sense. It seems tractible and would allow for other use cases (or maybe just simplify what can be done already on the command line) such as duplicating the same test file multiple times, or using an evergreen patch run's manifest as a starting point and pruning it to the tests one likes.

The simple thing I want to accomplish with this ticket:

Allow for a manifest file of the following format:

test_file1
test_file2
test_file1
test_file1

In this format there, any "job" (worker) can process a test file. There's no guarantee a single worker will process the same tests in repeated runs. This is effectively the same as listing out the files on the command line invocation.

The bonus thing to accomplish:
Allow for manifest file that can written in a second format. This is what an evergreen patch/commit run would output and upload as an artifact:

1 test_file1
2 test_file2
3 test_file3
2 test_file4
2 test_file5
1 test_file6
3 test_file7

The first column (number) is the job/worker that processed a test file. When consuming this manifest file, there's a guarantee that workers will work on the same tests in the same order. There's no guarantee that the actual tests being processed concurrently are the same between an evergreen patch/commit run and a local run.

The worker values can be given in any order to ease how a run outputs its manifest file. E.g: A stable sort on the first field will result in an equivalent execution.

Other goals:

  • Have resmoke allow and interpret a CLI argument @manifest_filename. This is a technique used by some programs invoked from the CLI that often take as input a long list of files (see javac):

    resmoke --suites concurrency_replication @testfiles # resmoke will read the contents of `testfiles` that is in one of the two described formats
    

Generated at Thu Feb 08 05:18:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.