Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-63565

Add task to validate runtime of tasks and tests

    • Fully Compatible
    • STM 2022-02-21
    • 2

      Use the Evergreen API to get the runtime and list of test that had run in burn-in, if any test is taking more than 1 standard deviation more than its historical average over the past 20 runs (or days, whichever is easier), fail the task with a helpful error message

      Determine if this approach is sufficient for reducing the runtime or if TIPS tooling is needed for things like change points.

      Design sketch:

      • After burn-in calls resmoke to run the tests, get the report object from resmoke and get the average timing of the tests
      • Call the Evergreen API to get the test timing for the past 20 days or runs
      • Raise exception if the average test time in burn in is greater than 1 stddev above the past average.

            mikhail.shchatko@mongodb.com Mikhail Shchatko
            robert.guo@mongodb.com Robert Guo (Inactive)
            0 Vote for this issue
            3 Start watching this issue