[SERVER-29062] Add support for silencing test failures in the mainline from specific tests Created: 04/May/17  Updated: 30/Oct/23  Resolved: 08/Jun/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.5.9

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Eddie Louie
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-29061 Add buildscripts/promote_silent_failu... Closed
depends on SERVER-29060 Add etc/test_retrial.yml resmoke.py t... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-05-29, TIG 2017-06-19
Participants:

 Description   

When not running a patch build, the "run tests" function in the etc/evergreen.yml project configuration should run resmoke.py once with

--suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
--tagFile=etc/test_retrial.yml
--excludeWithAnyTags=retry_on_failure,
                     retry_on_failure|${task_name},
                     retry_on_failure|${task_name}|${build_variant},
                     retry_on_failure|${task_name}|${build_variant}|${distro_id}
--reportFailureStatus=fail
--reportFile=report_reliable.json

and a second time with

--suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
--tagFile=etc/test_retrial.yml
--includeWithAnyTags=retry_on_failure,
                     retry_on_failure|${task_name},
                     retry_on_failure|${task_name}|${build_variant},
                     retry_on_failure|${task_name}|${build_variant}|${distro_id}
--reportFailureStatus=silentfail
--reportFile=report_unreliable.json
--continueOnFailure
--repeat=2

The promote_silent_failures.py script should be run on report_unreliable.json making it so that only if both executions of a test with an "retry_on_failure*" tag fail, would the task be marked as a failure.

Note: When the report_reliable.json is merged with the other report file using the combine_reports.py script, care should be taken to ensure the same test execution won't appear in the Evergreen sidebar multiple times as the combine_reports.py script will also be invoked as part of the "post" task with the changes from SERVER-28784.


Credit to adam.martin for this idea as a way to reduce the number of Evergreen task failures that the Build Baron needs to triage.



 Comments   
Comment by Githook User [ 08/Jun/17 ]

Author:

{u'username': u'elouie99', u'name': u'Eddie Louie', u'email': u'eddie.louie@mongodb.com'}

Message: SERVER-29062 Add support for silencing test failures in the mainline from specific tests
Branch: master
https://github.com/mongodb/mongo/commit/f7badec688af9d7488241f256df146a47d37759d

Comment by Eddie Louie [ 05/Jun/17 ]

Code review: https://mongodbcr.appspot.com/131860002/
Patch build: Evergreen patch

Generated at Thu Feb 08 04:19:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.