[SERVER-21030] Run changed/added tests 10x on patch builds Created: 20/Oct/15  Updated: 05/Apr/17  Resolved: 17/Jun/16

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Ian Whalen (Inactive) Assignee: Michael Grundy
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-21841 Add mode to resmoke.py to list under ... Closed
Duplicate
duplicates SERVER-24537 Create a new and modified test runnin... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 12 (04/01/16), TIG 16 (06/24/16)
Participants:

 Description   

Change to resmoke.py based on discussions with mpobrien and dan@10gen.com. The goal is that whenever an engineer runs a patch build that changes or adds a jstest, the patch build will automatically run with --repeat=10 for that test.



 Comments   
Comment by Kamran K. [ 17/Jun/16 ]

This functionality was added as part of SERVER-24537. We're only running with --repeat=2 for now, though.

Comment by Max Hirschhorn [ 10/Dec/15 ]

Filed SERVER-21841 to track a feature in resmoke.py to list the suites a test is run under. This can be used by an external tool to generate resmoke.py YAML configurations or invocations.

Moving this ticket to the server tools team backlog because the remaining components would be (1) extracting the changed tests from the patch applied, (2) mapping resmoke.py's suites to Evergreen tasks from the etc/evergreen.yml project configuration, and (3) generating resmoke.py invocations to run with --repeat=10 as Charlie described.

Comment by Charlie Swanson [ 22/Oct/15 ]

martin.bligh what are you referring to by 'there'? Could you move the discussion to the relevant ticket? I don't think this has much to do with this ticket anymore.

Comment by Martin Bligh [ 22/Oct/15 ]

Maybe we can just break out the logs there too?

Comment by Charlie Swanson [ 22/Oct/15 ]

max.hirschhorn, martin.bligh's point about it being hard to read the logs locally is a good one. I have often run into this issue, where I run things in parallel to test my changes, have something fail, and then have to re-run just that one test to figure out why it's failing. I've filed SERVER-21073 and SERVER-21074 to improve the workflow of using many jobs to run a large suite locally. We had already discussed the latter, but I couldn't find a ticket, so I created one.

Comment by Martin Bligh [ 22/Oct/15 ]

Ah, OK, I was looking at local logs - good point

Comment by Eric Milkie [ 22/Oct/15 ]

But it's not like compilation with -j, because each parallel stream keeps a separate log, which is accessible via the Evergreen UI. (It's true that if you make use of -j parallelism to run tests with resmoke on your local machine, the logs by default all go to the same place and interleave.)

Comment by Martin Bligh [ 22/Oct/15 ]

FWIW, now that we started running a lot of tests in parallel, it seems like it's going to be a lot more difficult to see the error messages from a failing test (just like compilation with -j) ... not sure if it's also possible to repeat the test(s) that failed single threaded at the end.

I don't think that ties into this ticket, since I suspect you have a lot more context inside resmoke already at this point, but thought I'd mention it now in case.

Comment by Eric Milkie [ 22/Oct/15 ]

We cannot derive the test suites from the relative path of any .js file because any jstest file can be referenced by any suite in any .yml config file (which is how it should be).
As Charlie said, we will need to scan all the suites to see which ones contain the changed jstest files, and then run "something" that only tests the changed files in each suite. However, I don't think the Evergreen interface is prepared for this; there may be a lot of work to do to make the output from this "fit" into the current UI for a patch build and still have all the links to logs and error detection work sanely.

Comment by Martin Bligh [ 22/Oct/15 ]

Can we derive the tests suites from the relative path of the .js file in an automated fashion?
It doesn't seem like the directory names, the evergreen test names, and the test suite names are consistent, which is unfortunate - makes it harder for manual repro too.
Maybe we can come up with a rule? I presume it's 1-to-many from dir name to suite name though

Comment by Michael O'Brien [ 22/Oct/15 ]

What sort of changes to resmoke.py are we envisioning? Actually doing the git diff to see which tests changed? That seems like an odd responsibility for resmoke.py to have.

Logic for this would definitely be external, it would be gnarly for resmoke to have to touch anything involving git.
We were basically just looking for the ability to override the "repeat" flag for particular tests only. If the behavior can already be accomplished using what you stated in #3 then we're already good here

Comment by Charlie Swanson [ 22/Oct/15 ]

What sort of changes to resmoke.py are we envisioning? Actually doing the git diff to see which tests changed? That seems like an odd responsibility for resmoke.py to have.

I don't think it would be hard for some logic external to resmoke.py to figure it out. Either evergreen could do it, or we could write some fun bash script in our evergreen.yml to do something like the following before or after running any given suite:

  1. Look at the patch/diff to see which test files are affected
  2. List the tests in that suite to check if the affected test(s) is/are part of the suite, something like

    python buildscripts/resmoke.py --suite=<suite> -n | grep path/to/changed/test.js
    

    for each test

  3. If the test(s) is/are part of the suite, it could run the following after the suite to accomplish the desired behavior:

    python buildscripts/resmoke.py --executor=<suite> path/to/changed/test1.js path/to/changed/test2.js --repeat=10 --continueOnFailure <other options, e.g. buildlogger setup>
    

Comment by Ian Whalen (Inactive) [ 22/Oct/15 ]

Pretty certain that the goal is to make it a bit more intelligent than that and only run the specific jstest files 10x.

Comment by Eric Milkie [ 20/Oct/15 ]

Since patch builds run test suites, not individual jstest files, did you mean to say that the entire suite will run with --repeat=10, or is this more involved than that?

Generated at Thu Feb 08 03:56:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.