Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-29062

Add support for silencing test failures in the mainline from specific tests

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.5.9
    • Component/s: Testing Infrastructure
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Sprint:
      TIG 2017-05-29, TIG 2017-06-19

      Description

      When not running a patch build, the "run tests" function in the etc/evergreen.yml project configuration should run resmoke.py once with

      --suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
      --tagFile=etc/test_retrial.yml
      --excludeWithAnyTags=retry_on_failure,
                           retry_on_failure|${task_name},
                           retry_on_failure|${task_name}|${build_variant},
                           retry_on_failure|${task_name}|${build_variant}|${distro_id}
      --reportFailureStatus=fail
      --reportFile=report_reliable.json
      

      and a second time with

      --suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
      --tagFile=etc/test_retrial.yml
      --includeWithAnyTags=retry_on_failure,
                           retry_on_failure|${task_name},
                           retry_on_failure|${task_name}|${build_variant},
                           retry_on_failure|${task_name}|${build_variant}|${distro_id}
      --reportFailureStatus=silentfail
      --reportFile=report_unreliable.json
      --continueOnFailure
      --repeat=2
      

      The promote_silent_failures.py script should be run on report_unreliable.json making it so that only if both executions of a test with an "retry_on_failure*" tag fail, would the task be marked as a failure.

      Note: When the report_reliable.json is merged with the other report file using the combine_reports.py script, care should be taken to ensure the same test execution won't appear in the Evergreen sidebar multiple times as the combine_reports.py script will also be invoked as part of the "post" task with the changes from SERVER-28784.


      Credit to ADAM Martin for this idea as a way to reduce the number of Evergreen task failures that the Build Baron needs to triage.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              eddie.louie Eddie Louie
              Reporter:
              max.hirschhorn Max Hirschhorn
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: