Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-63565

Add task to validate runtime of tasks and tests

    • Fully Compatible
    • STM 2022-02-21
    • 2

      Use the Evergreen API to get the runtime and list of test that had run in burn-in, if any test is taking more than 1 standard deviation more than its historical average over the past 20 runs (or days, whichever is easier), fail the task with a helpful error message

      Determine if this approach is sufficient for reducing the runtime or if TIPS tooling is needed for things like change points.

      Design sketch:

      • After burn-in calls resmoke to run the tests, get the report object from resmoke and get the average timing of the tests
      • Call the Evergreen API to get the test timing for the past 20 days or runs
      • Raise exception if the average test time in burn in is greater than 1 stddev above the past average.

            Assignee:
            mikhail.shchatko@mongodb.com Mikhail Shchatko
            Reporter:
            robert.guo@mongodb.com Robert Guo (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: