[SERVER-34593] resmoke.py should be able to run multiple instances of a single test in parallel Created: 20/Apr/18  Updated: 29/Oct/23  Resolved: 29/Aug/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.1.3

Type: Improvement Priority: Major - P3
Reporter: William Schultz (Inactive) Assignee: Yves Duhem
Resolution: Fixed Votes: 1
Labels: tig-qwin-eligible, tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-40671 Resmoke does not parallelize executio... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2018-09-10
Participants:
Story Points: 3

 Description   

If you try to run a single Javascript test with resmoke.py using a combination of the --repeat=N flag and the -j=M flag, it will still run the test sequentially. e.g.

python buildscripts/resmoke.py --repeat=100 -j10 sometest.js

Ideally it could parallelize repeated execution of a single test. For example, if --repeat=100 and -j=10, it would run 10 instances of the test in parallel, that would each execute 10 times.

This could be very helpful for quickly trying to reproduce a particular test failure locally.



 Comments   
Comment by Githook User [ 29/Aug/18 ]

Author:

{'name': 'Yves Duhem', 'email': 'yves.duhem@mongodb.com', 'username': 'syev'}

Message: SERVER-34593 New resmoke option to repeat tests
Branch: master
https://github.com/mongodb/mongo/commit/d92fe6cd9242a22e8ae56f48e64a20770d9e8291

Comment by William Schultz (Inactive) [ 23/Jul/18 ]

Sounds good to me.

Comment by Max Hirschhorn [ 23/Jul/18 ]

I think the plan was to introduce --repeatTests and then eventually rename --repeat to --repeatSuites. Yves picked a similar name because they are related

Comment by William Schultz (Inactive) [ 23/Jul/18 ]

Ok. So then --repeatTests=N would act to create N "virtual" instances of the given test(s), within a suite, as I understand it. The approach seems reasonable, though maybe calling the new argument something like --duplicateTests might help reduce confusion since --repeatTests sounds similar to the existing --repeat argument. That can be discussed in a CR, though.

Comment by Max Hirschhorn [ 23/Jul/18 ]

I think that seems fine. If I run

python buildscripts/resmoke.py sometest.js

is it treated as if I am running a single suite, with one test in it, even though I didn't actually specify a particular suite?

resmoke.py defaults to use the with_server.yml YAML suite configuration when you omit the --suites command line option. So there is always a particular suite even if you don't specify it explicitly.

Comment by William Schultz (Inactive) [ 23/Jul/18 ]

I think that seems fine. If I run

python buildscripts/resmoke.py sometest.js

is it treated as if I am running a single suite, with one test in it, even though I didn't actually specify a particular suite?

Comment by Yves Duhem [ 18/Jul/18 ]

The --repeat parameter is used to repeat suites (the suites are run sequentially) and the --jobs parameter defines the number of jobs that are used to run the tests in a suite.
In order to repeat a test inside a suite (so it can take advantage of the parallelization), we can introduce a new parameter --repeatTests that would repeat the tests inside the suite.
william.schultz, your example would then be:

python buildscripts/resmoke.py -j10 --repeatTests=100 sometest.js

Comment by Max Hirschhorn [ 23/Apr/18 ]

I think you can cheat the system and achieve this by passing the test multiple times on the command line:

While this used to work, I suspect some changes to the selector.py module has caused resmoke.py to convert the list of test files into a set instance earlier than it did before.

Comment by William Schultz (Inactive) [ 23/Apr/18 ]

I don't think that actually parallelizes the test execution. For example, if you run:

python buildscripts/resmoke.py --basePort=35000 --suite=core --repeat=16 -j4 jstests/core/all2.js jstests/core/all2.js jstests/core/all2.js jstests/core/all2.js

how many mongod processes are actually spun up on the machine? (ps -ax | grep "mongod.*35). For me I only see 1 mongod. If, however, I run

python buildscripts/resmoke.py --basePort=35000 --suite=core --repeat=16 -j4 jstests/core/all.js jstests/core/all2.js jstests/core/all3.js jstests/core/all4.js

I see 4 mongod processes.

Comment by Charlie Swanson [ 23/Apr/18 ]

I think you can cheat the system and achieve this by passing the test multiple times on the command line:

python buildscripts/resmoke.py --repeat=100 -j10 sometest.js sometest.js sometest.js sometest.js sometest.js sometest.js sometest.js sometest.js sometest.js sometest.js

Generated at Thu Feb 08 04:37:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.