-
Type:
New Feature
-
Resolution: Fixed
-
Priority:
Major - P3
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
Labels:None
-
Fully Compatible
-
TIG 2017-05-29, TIG 2017-06-19
When not running a patch build, the "run tests" function in the etc/evergreen.yml project configuration should run resmoke.py once with
--suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
--tagFile=etc/test_retrial.yml
--excludeWithAnyTags=retry_on_failure,
retry_on_failure|${task_name},
retry_on_failure|${task_name}|${build_variant},
retry_on_failure|${task_name}|${build_variant}|${distro_id}
--reportFailureStatus=fail
--reportFile=report_reliable.json
and a second time with
--suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
--tagFile=etc/test_retrial.yml
--includeWithAnyTags=retry_on_failure,
retry_on_failure|${task_name},
retry_on_failure|${task_name}|${build_variant},
retry_on_failure|${task_name}|${build_variant}|${distro_id}
--reportFailureStatus=silentfail
--reportFile=report_unreliable.json
--continueOnFailure
--repeat=2
The promote_silent_failures.py script should be run on report_unreliable.json making it so that only if both executions of a test with an "retry_on_failure*" tag fail, would the task be marked as a failure.
Note: When the report_reliable.json is merged with the other report file using the combine_reports.py script, care should be taken to ensure the same test execution won't appear in the Evergreen sidebar multiple times as the combine_reports.py script will also be invoked as part of the "post" task with the changes from SERVER-28784.
Credit to adam.martin for this idea as a way to reduce the number of Evergreen task failures that the Build Baron needs to triage.
- depends on
-
-
- Closed
-
-
SERVER-29060 Add etc/test_retrial.yml resmoke.py tag file
-
- Closed
-