[SERVER-63565] Add task to validate runtime of tasks and tests Created: 11/Dec/21  Updated: 29/Oct/23  Resolved: 21/Feb/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.0-rc0

Type: Task Priority: Major - P3
Reporter: Robert Guo (Inactive) Assignee: Mikhail Shchatko
Resolution: Fixed Votes: 0
Labels: faster-patches, tig-qwin-eligible
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Sprint: STM 2022-02-21
Participants:
Story Points: 2

 Description   

Use the Evergreen API to get the runtime and list of test that had run in burn-in, if any test is taking more than 1 standard deviation more than its historical average over the past 20 runs (or days, whichever is easier), fail the task with a helpful error message

Determine if this approach is sufficient for reducing the runtime or if TIPS tooling is needed for things like change points.

Design sketch:

  • After burn-in calls resmoke to run the tests, get the report object from resmoke and get the average timing of the tests
  • Call the Evergreen API to get the test timing for the past 20 days or runs
  • Raise exception if the average test time in burn in is greater than 1 stddev above the past average.


 Comments   
Comment by Githook User [ 21/Feb/22 ]

Author:

{'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com', 'username': 'MikhailShchatko'}

Message: SERVER-63565 Validate tests runtime in burn-in
Branch: master
https://github.com/mongodb/mongo/commit/193a020de5257a91a72a38ea6d18d5ce05deed01

Generated at Thu Feb 08 05:58:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.