Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-34296

Create Python script to generate resmoke.py YAML suite files running subset of tests with target execution time

XMLWordPrintableJSON

    • Type: Icon: New Feature New Feature
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 3.7.4
    • Affects Version/s: None
    • Component/s: Testing Infrastructure
    • Labels:
      None
    • Fully Compatible
    • TIG 2018-04-23
    • 0

      Create a Python script that given (a) the name of an Evergreen task, (b) a list of build variants, and (c) a target execution time, uses Evergreen’s REST API to compute the time taken by recent successful executions of the test and generates resmoke.py YAML suite files running subsets of the tests taking no longer than the target execution time.

      • It should be possible to modify buildscripts/test_failures.py to report the duration of successful executions instead of just the number of their occurrences. Doing so would avoid needing to create another wrapper around the /test_history Evergreen endpoint.
      • The algorithm for determining how to split up a resmoke.py test suite should prefer to generate more resmoke.py YAML suite files that are well under the target execution time than to generate any resmoke.py YAML suite files that are above the target execution time given the historical average of each test's execution time.

      The Python script should create multiple resmoke.py YAML suite files in the buildscripts/resmokeconfig/suites/ directory along the lines of the following, with perhaps some additional information written to stdout describing what build variants the new Evergreen tasks should be run against.

      sharding_000.yml
      # This file was generated by buildscripts/generate_resmoke_suites.py and manual edits should also be
# made to sharding.yml in order to ensure running the full test suite locally matches the behavior
# of running parts of the test suite in Evergreen. It was generated against commit ...
# with the following options: ...
test_kind: js_test

selector: 
  # The following tests take approximately
  #   17 minutes to run on Enterprise RHEL 6.2
  #   23 minutes to run on Linux DEBUG
  #   34 minutes to run on Enterprise Windows 2008R2
  roots: 
  - jstests/sharding/aggregation_currentop.js
  - jstests/sharding/cleanup_orphaned_auth.js
  - jstests/sharding/crash_mongos_against_upgraded_cluster.js
  - jstests/sharding/cursor_valid_after_shard_stepdown.js
  - jstests/sharding/delete_during_migrate.js
  - jstests/sharding/mapReduce_nonSharded.js
  - jstests/sharding/migration_id_index.js
  - jstests/sharding/migration_ignore_interrupts_3.js
  - jstests/sharding/migration_server_status.js
  - jstests/sharding/min_optime_recovery_on_successful_move_chunk_commit.js
  - jstests/sharding/mongos_query_comment.js
  - jstests/sharding/mongos_shard_failure_tolerance.js
  - jstests/sharding/movechunk_with_default_paranoia.js
  - jstests/sharding/movechunk_with_moveParanoia.js
  - jstests/sharding/mr_shard_version.js
  - jstests/sharding/read_does_not_create_namespaces.js
  - jstests/sharding/shard_aware_init.js
  - jstests/sharding/shard_kill_and_pooling.js
  - jstests/sharding/tag_auto_split.js
  - jstests/sharding/tag_range.js

executor: 
  config: 
    shell_options: 
      nodb: ''
      readMode: commands

            Assignee:
            david.bradford@mongodb.com David Bradford (Inactive)
            Reporter:
            max.hirschhorn@mongodb.com Max Hirschhorn
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: