[SERVER-28784] Update "run tests" function to run unreliable tests separately Created: 12/Apr/17  Updated: 12/Oct/17  Resolved: 09/May/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.5.7

Type: Task Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jonathan Abrahams
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
Gantt Dependency
has to be done after SERVER-28783 Add buildscripts/combine_reports.py s... Closed
Related
related to SERVER-29306 C++ unit tests run twice in Evergreen... Closed
is related to SERVER-29507 Task not marked as failure if resmoke... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-05-08, TIG 2017-05-29
Participants:
Linked BF Score: 0

 Description   

When running a patch build, the "run tests" function in the etc/evergreen.yml project configuration should run resmoke.py once with

--suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
--tagFile=etc/test_lifecycle.yml
--excludeWithAnyTags=unreliable,
                     unreliable|${task_name},
                     unreliable|${task_name}|${build_variant},
                     unreliable|${task_name}|${build_variant}|${distro_id}
--reportFailureStatus=fail
--reportFile=report_reliable.json

and a second time with

--suites=<suite_name> --excludeWithAnyTags=<suite_tags_to_exclude> ...
--tagFile=etc/test_lifecycle.yml
--includeWithAnyTags=unreliable,
                     unreliable|${task_name},
                     unreliable|${task_name}|${build_variant},
                     unreliable|${task_name}|${build_variant}|${distro_id}
--reportFailureStatus=silentfail
--reportFile=report_unreliable.json

The exit code of the "run tests" function should be based on the exit code of the combine_reports.py script. The combine_reports.py script should also be run as part of the "post" task so that it happens even if the task times out since the SIGUSR1 and Windows event object should cause resmoke.py to write out a report.json file.

Note: The "unreliable*" tags should not be used when running in the Evergreen mainline.



 Comments   
Comment by Githook User [ 09/May/17 ]

Author:

{u'username': u'hptabster', u'name': u'Jonathan Abrahams', u'email': u'jonathan@mongodb.com'}

Message: SERVER-28784 burn_in_tests.py creates empty report if there are no new or modified tests
Branch: master
https://github.com/mongodb/mongo/commit/e351e669b05a82c0aea1d71538efc0cd0bf94dc4

Comment by Jonathan Abrahams [ 08/May/17 ]

We need to handle 2 more cases:

  • No new tests are discovered by buildscripts/burn_in_tests.py
  • Handle quoting and curly brace in test_flags in the variant rollback-3dot4-rhel-62
Comment by Githook User [ 05/May/17 ]

Author:

{u'username': u'hptabster', u'name': u'Jonathan Abrahams', u'email': u'jonathan@mongodb.com'}

Message: SERVER-28784 Update "run tests" function to run unreliable tests separately
Branch: master
https://github.com/mongodb/mongo/commit/3254737dde79507d1f1c075e41e51559e59a426c

Generated at Thu Feb 08 04:19:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.