A test should be added to the set of tests believed not to run reliably when it has more than 2·min_runs executions in the last 4 weeks and a failure percentage greater than unacceptable_fail_rate. A test should then removed from the set of tests believed not to run reliably when it has at least min_runs executions in the last 2 weeks and a failure percentage less than acceptable_fail_rate, where acceptable_fail_rate < unacceptable_fail_rate.

For each of the k-tuples mentioned below, an independent unacceptable_fail_rate_k and acceptable_fail_rate_k will be chosen by analyzing historical failures in Evergreen. The model described above will then be applied to

• The 1-tuple (test name) to produce the unreliable tag,
• The 2-tuple (test name, task name) to produce the unreliable|${task_name} tag,
• The 3-tuple (test name, task name, build variant) to produce the unreliable|${task_name}|${build_variant} tag, and
• The 4-tuple (test name, task name, build variant, distro) to produce the unreliable|${task_name}|${build_variant}|${distro_id} tag.

The Python script to compute the test failure rates using the Evergreen API from SERVER-28782 should be integrated into update_test_lifecycle.py script. The update_test_lifecycle.py script should then overwrite the contents of the etc/test_lifecycle.yml file in the mongodb/mongo repository. Follow-up work on the update_test_lifecycle.py script will involve automatically committing these changes and filing new JIRA tickets in the SERVER project.

The update_test_lifecycle.py script should ensure the keys and the values contained in lists of the etc/test_lifecycle.yml file are sorted so the generated diffs are readable to a human and the output doesn't change if the tool is immediately rerun.