[SERVER-21030] Run changed/added tests 10x on patch builds Created: 20/Oct/15 Updated: 05/Apr/17 Resolved: 17/Jun/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Ian Whalen (Inactive) | Assignee: | Michael Grundy |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Sprint: | TIG 12 (04/01/16), TIG 16 (06/24/16) | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
Change to resmoke.py based on discussions with mpobrien and dan@10gen.com. The goal is that whenever an engineer runs a patch build that changes or adds a jstest, the patch build will automatically run with --repeat=10 for that test. |
| Comments |
| Comment by Kamran K. [ 17/Jun/16 ] | ||
|
This functionality was added as part of | ||
| Comment by Max Hirschhorn [ 10/Dec/15 ] | ||
|
Filed Moving this ticket to the server tools team backlog because the remaining components would be (1) extracting the changed tests from the patch applied, (2) mapping resmoke.py's suites to Evergreen tasks from the etc/evergreen.yml project configuration, and (3) generating resmoke.py invocations to run with --repeat=10 as Charlie described. | ||
| Comment by Charlie Swanson [ 22/Oct/15 ] | ||
|
martin.bligh what are you referring to by 'there'? Could you move the discussion to the relevant ticket? I don't think this has much to do with this ticket anymore. | ||
| Comment by Martin Bligh [ 22/Oct/15 ] | ||
|
Maybe we can just break out the logs there too? | ||
| Comment by Charlie Swanson [ 22/Oct/15 ] | ||
|
max.hirschhorn, martin.bligh's point about it being hard to read the logs locally is a good one. I have often run into this issue, where I run things in parallel to test my changes, have something fail, and then have to re-run just that one test to figure out why it's failing. I've filed | ||
| Comment by Martin Bligh [ 22/Oct/15 ] | ||
|
Ah, OK, I was looking at local logs - good point | ||
| Comment by Eric Milkie [ 22/Oct/15 ] | ||
|
But it's not like compilation with -j, because each parallel stream keeps a separate log, which is accessible via the Evergreen UI. (It's true that if you make use of -j parallelism to run tests with resmoke on your local machine, the logs by default all go to the same place and interleave.) | ||
| Comment by Martin Bligh [ 22/Oct/15 ] | ||
|
FWIW, now that we started running a lot of tests in parallel, it seems like it's going to be a lot more difficult to see the error messages from a failing test (just like compilation with -j) ... not sure if it's also possible to repeat the test(s) that failed single threaded at the end. I don't think that ties into this ticket, since I suspect you have a lot more context inside resmoke already at this point, but thought I'd mention it now in case. | ||
| Comment by Eric Milkie [ 22/Oct/15 ] | ||
|
We cannot derive the test suites from the relative path of any .js file because any jstest file can be referenced by any suite in any .yml config file (which is how it should be). | ||
| Comment by Martin Bligh [ 22/Oct/15 ] | ||
|
Can we derive the tests suites from the relative path of the .js file in an automated fashion? | ||
| Comment by Michael O'Brien [ 22/Oct/15 ] | ||
Logic for this would definitely be external, it would be gnarly for resmoke to have to touch anything involving git. | ||
| Comment by Charlie Swanson [ 22/Oct/15 ] | ||
|
What sort of changes to resmoke.py are we envisioning? Actually doing the git diff to see which tests changed? That seems like an odd responsibility for resmoke.py to have. I don't think it would be hard for some logic external to resmoke.py to figure it out. Either evergreen could do it, or we could write some fun bash script in our evergreen.yml to do something like the following before or after running any given suite:
| ||
| Comment by Ian Whalen (Inactive) [ 22/Oct/15 ] | ||
|
Pretty certain that the goal is to make it a bit more intelligent than that and only run the specific jstest files 10x. | ||
| Comment by Eric Milkie [ 20/Oct/15 ] | ||
|
Since patch builds run test suites, not individual jstest files, did you mean to say that the entire suite will run with --repeat=10, or is this more involved than that? |