[SERVER-28782] Write Python script to compute test failures rates from Evergreen API Created: 12/Apr/17  Updated: 12/Oct/17  Resolved: 24/May/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.5.8

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Jonathan Abrahams
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-28786 Add buildscripts/update_test_lifecycl... Closed
Related
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-05-08, TIG 2017-05-29
Participants:

 Description   

A /projects/{project_id}/test_history API endpoint was added as part of EVG-1378 that returns information about test failures from Evergreen task failures. This endpoint should be used to retrieve per-test execution information of Evergreen tasks that have run in the last 4 weeks. A possible command line interface for this script may look like the following:

--since=<date_since>        (defaults to 1 week ago)
--until=<date_until>        (defaults to now)
--group=[daily|weekly]      (defaults to "weekly")
--task=<task_display_name>
--variant=<variant_id>
--distro=<distro_id>
test1 [test2 ...]

where the output of the script is the per-day failure rate or the per-week failure rate for each (test name, task name, build variant, distro) combination. In addition to outputting, the Python script should be able to be used as a Python module in order to be able to update etc/test_lifecycle.yml as part of SERVER-28786.

Additionally, the Python script should attempt to read the Evergreen API credentials from ~/.evergreen.yml if the file is present.

Note: Since the script needs to compute the ratio of (fail + timeout) / (fail + timeout + pass), a large number of API requests may need to be made to get the entire test execution history of a 4 week period. We'll need to consider a different approach or possible optimizations of only examining tests that have failed at least once before retrieving the entire test execution history.



 Comments   
Comment by Githook User [ 24/May/17 ]

Author:

{u'username': u'hptabster', u'name': u'Jonathan Abrahams', u'email': u'jonathan@mongodb.com'}

Message: SERVER-28782 Add buildscripts/test_failures.py, script to compute test failures rates from Evergreen API
Branch: master
https://github.com/mongodb/mongo/commit/c609be45647fce98d0394221efc5d362ac470b64

Generated at Thu Feb 08 04:19:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.