[SERVER-39313] Create burn_in_tests metric tracking script Created: 31/Jan/19  Updated: 29/Oct/23  Resolved: 02/Apr/19

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.1.10

Type: New Feature Priority: Major - P3
Reporter: Jonathan Abrahams Assignee: Jonathan Abrahams
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: DAG 2019-03-25, DAG 2019-04-08
Participants:
Story Points: 3

 Description   

A script to track the effectiveness of burn_in_tests will be manually run. This script should be called buildscripts/metrics/burn_in_tests.py.

The script will invoke the task history of patch and mainline builds over a user specific period (default 4 weeks). It will provide the following, for builds which ran burn_in_tests:

  • Number of patch builds
  • Number of failing tasks
  • Number of failing burn_in_tests tasks
  • Number of patch builds where only burn_in_tests failed
  • Number of tasks generated
  • Number of tests executed
  • Number of times task exceeded the expected run time

AWS costs
Computing the AWS costs for associated to each burn_in task (and the sub-tasks it spawned) will be useful in understanding what additional cost there is to run burn_in more than twice. These costs can be computed using task history time for all generated tasks and the main burn_in task weighted with the type of distro the tasks ran on. Contrasting this result with prior burn_in tasks runs will provide some metric for the increased cost.

The aws costs for burn_in can be computed with the following splunk query:
index=evergreen stat = task-end-stats task = burn_in_tests* project = mongodb-mongo-master | timechart span=1h avg(cost)



 Comments   
Comment by Githook User [ 02/Apr/19 ]

Author:

{'name': 'Jonathan Abrahams', 'username': 'hptabster', 'email': 'jonathan@mongodb.com'}

Message: SERVER-39313 Metric tracking script for burn_in_tests
Branch: master
https://github.com/mongodb/mongo/commit/4b5ea6e22404c5ebb5208bd96fad69ba2fa0cdb8

Generated at Thu Feb 08 04:51:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.