Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-28784

Update "run tests" function to run unreliable tests separately

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.5.7
    • Component/s: Testing Infrastructure
    • Labels:
      None
    • Backwards Compatibility:
      Fully Compatible
    • Epic Link:
    • Sprint:
      TIG 2017-05-08, TIG 2017-05-29
    • Linked BF Score:
      0

      Description

      When running a patch build, the "run tests" function in the etc/evergreen.yml project configuration should run resmoke.py once with

      --suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
      --tagFile=etc/test_lifecycle.yml
      --excludeWithAnyTags=unreliable,
                           unreliable|${task_name},
                           unreliable|${task_name}|${build_variant},
                           unreliable|${task_name}|${build_variant}|${distro_id}
      --reportFailureStatus=fail
      --reportFile=report_reliable.json
      

      and a second time with

      --suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
      --tagFile=etc/test_lifecycle.yml
      --includeWithAnyTags=unreliable,
                           unreliable|${task_name},
                           unreliable|${task_name}|${build_variant},
                           unreliable|${task_name}|${build_variant}|${distro_id}
      --reportFailureStatus=silentfail
      --reportFile=report_unreliable.json
      

      The exit code of the "run tests" function should be based on the exit code of the combine_reports.py script. The combine_reports.py script should also be run as part of the "post" task so that it happens even if the task times out since the SIGUSR1 and Windows event object should cause resmoke.py to write out a report.json file.

      Note: The "unreliable*" tags should not be used when running in the Evergreen mainline.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: