Add task to validate runtime of tasks and tests

XMLWordPrintableJSON

    • Fully Compatible
    • STM 2022-02-21
    • 2
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Use the Evergreen API to get the runtime and list of test that had run in burn-in, if any test is taking more than 1 standard deviation more than its historical average over the past 20 runs (or days, whichever is easier), fail the task with a helpful error message

      Determine if this approach is sufficient for reducing the runtime or if TIPS tooling is needed for things like change points.

      Design sketch:

      • After burn-in calls resmoke to run the tests, get the report object from resmoke and get the average timing of the tests
      • Call the Evergreen API to get the test timing for the past 20 days or runs
      • Raise exception if the average test time in burn in is greater than 1 stddev above the past average.

              Assignee:
              Mikhail Shchatko
              Reporter:
              Robert Guo (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: