[SERVER-21074] Add ability to resmoke.py to pick up where you left off running a suite Created: 22/Oct/15  Updated: 06/Dec/22  Resolved: 19/Nov/21

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 3.2.0-rc0
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Charlie Swanson Assignee: Backlog - Server Tooling and Methods (STM) (Inactive)
Resolution: Won't Fix Votes: 1
Labels: tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Server Tooling & Methods
Participants:

 Description   

When running a large suite locally to test your changes, the following is a common work flow

python buildscripts/resmoke.py --suites=<suite> -j<N>

You notice a test fails, and you figure out why, fix it, then try again

python buildscripts/resmoke.py --suites=<suite> -j<N>

Then, after repeating the execution of all the other tests, some other test fails, so you repeat the whole process.

This all can be very time consuming. In many cases, you don't need to re-run all the tests that succeeded the first time. Either the fix was to the specific test, or for some other reason you are confident that they won't start failing due to your change.

If there was a report.json file present, resmoke.py could use this file to determine which tests in the suite failed or were skipped last time, and only run those. This behavior should probably be behind some flag, since the user would have to know it was 'safe' not to re-run the tests that previously succeeded. As part of this change, it would probably be useful to generate a report.json file on each invocation of resmoke.py, so the user doesn't need to have the foresight to include this option.



 Comments   
Comment by Mathias Stearn [ 31/May/17 ]

This would need to distinguish two meanings of "skipped" in the report.json: skipped because an earlier test failed vs skipped because it was run in an earlier pass.

Generated at Thu Feb 08 03:56:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.