[SERVER-28786] Add buildscripts/update_test_lifecycle.py script for updating etc/test_lifecycle.yml Created: 13/Apr/17  Updated: 12/Oct/17  Resolved: 15/Jun/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.5.9

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jonathan Abrahams
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-28782 Write Python script to compute test f... Closed
depends on SERVER-29486 resmoke method create_test_membership... Closed
Gantt Dependency
Related
is related to SERVER-29821 Modify algorithm to update etc/test_l... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-05-29, TIG 2017-06-19
Participants:

 Description   

A test should be added to the set of tests believed not to run reliably when it has more than 2ยทmin_runs executions in the last 4 weeks and a failure percentage greater than unacceptable_fail_rate. A test should then removed from the set of tests believed not to run reliably when it has at least min_runs executions in the last 2 weeks and a failure percentage less than acceptable_fail_rate, where acceptable_fail_rate < unacceptable_fail_rate.

For each of the k-tuples mentioned below, an independent unacceptable_fail_ratek and acceptable_fail_ratek will be chosen by analyzing historical failures in Evergreen. The model described above will then be applied to

  • The 1-tuple (test name) to produce the unreliable tag,
  • The 2-tuple (test name, task name) to produce the unreliable|${task_name} tag,
  • The 3-tuple (test name, task name, build variant) to produce the unreliable|${task_name}|${build_variant} tag, and
  • The 4-tuple (test name, task name, build variant, distro) to produce the unreliable|${task_name}|${build_variant}|${distro_id} tag.

The Python script to compute the test failure rates using the Evergreen API from SERVER-28782 should be integrated into update_test_lifecycle.py script. The update_test_lifecycle.py script should then overwrite the contents of the etc/test_lifecycle.yml file in the mongodb/mongo repository. Follow-up work on the update_test_lifecycle.py script will involve automatically committing these changes and filing new JIRA tickets in the SERVER project.

The update_test_lifecycle.py script should ensure the keys and the values contained in lists of the etc/test_lifecycle.yml file are sorted so the generated diffs are readable to a human and the output doesn't change if the tool is immediately rerun.



 Comments   
Comment by Githook User [ 15/Jun/17 ]

Author:

{u'username': u'hptabster', u'name': u'Jonathan Abrahams', u'email': u'jonathan@mongodb.com'}

Message: SERVER-28786 Add buildscripts/update_test_lifecycle.py script for updating etc/test_lifecycle.yml
Branch: master
https://github.com/mongodb/mongo/commit/75a15b5e7c195af9db19ae995867e63a8f89f861

Generated at Thu Feb 08 04:19:03 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.