Uploaded image for project: 'WiredTiger'
  1. WiredTiger
  2. WT-6097

Let the Python test suite runner allocate tests to buckets for evergreen

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • WT10.0.0, 4.4.0-rc5, 4.7.0
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None
    • 3
    • Storage - Tora 2020-05-18

      This came out of talking with luke.chen about WT-5801.  It would be straightforward for the run.py program to break down the list of tests into buckets of equitable size.  Here's how it works currently in evergreen.  We have 11 buckets, bucket 00 runs "python run.py [a|b]" to match tests that start with 'a' or 'b'.  Other buckets break down the alphabet - but since some letters have a lopsided number of slower tests, they are broken down into smaller subdivision.

      Instead, consider run.py with a new -b option.  Bucket 00 runs python run.py -b 0/11 (that is, bucket 0 of 11), bucket 01 runs python run.py -b 1/11 (that is, bucket 1 of 11).  What run.py does internally when it sees -b is create the entire list (it already can do tihs for the -dryrun option).  It "deals" them out into 11 piles (like one deals cards) and for 0/11, it takes the 0'th pile.  Since we deal the tests, tests of similar naming get put into different piles, and we should get a set of piles that are pretty equal.  A lot more equal than managing an alphabetic breakdown. It's not perfect (something like a work queue would be best), but with 12000 separate test and scenarios that tend to run sub-second, it's likely that the bucket times will come out somewhat even.

            Assignee:
            donald.anderson@mongodb.com Donald Anderson
            Reporter:
            donald.anderson@mongodb.com Donald Anderson
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: