A script to track the effectiveness of burn_in_tests will be manually run. This script should be called buildscripts/metrics/burn_in_tests.py.

The script will invoke the task history of patch and mainline builds over a user specific period (default 4 weeks). It will provide the following, for builds which ran burn_in_tests:

• Number of patch builds
• Number of failing tasks
• Number of failing burn_in_tests tasks
• Number of patch builds where only burn_in_tests failed
• Number of tasks generated
• Number of tests executed
• Number of times task exceeded the expected run time

AWS costs
Computing the AWS costs for associated to each burn_in task (and the sub-tasks it spawned) will be useful in understanding what additional cost there is to run burn_in more than twice. These costs can be computed using task history time for all generated tasks and the main burn_in task weighted with the type of distro the tasks ran on. Contrasting this result with prior burn_in tasks runs will provide some metric for the increased cost.

The aws costs for burn_in can be computed with the following splunk query:
index=evergreen stat = task-end-stats task = burn_in_tests* project = mongodb-mongo-master | timechart span=1h avg(cost)