[SERVER-34296] Create Python script to generate resmoke.py YAML suite files running subset of tests with target execution time Created: 04/Apr/18  Updated: 29/Oct/23  Resolved: 16/Apr/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.7.4

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: David Bradford (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-34330 Split up long-running Evergreen tasks... Closed
Related
Backwards Compatibility: Fully Compatible
Sprint: TIG 2018-04-23
Participants:
Linked BF Score: 0

 Description   

Create a Python script that given (a) the name of an Evergreen task, (b) a list of build variants, and (c) a target execution time, uses Evergreen’s REST API to compute the time taken by recent successful executions of the test and generates resmoke.py YAML suite files running subsets of the tests taking no longer than the target execution time.

  • It should be possible to modify buildscripts/test_failures.py to report the duration of successful executions instead of just the number of their occurrences. Doing so would avoid needing to create another wrapper around the /test_history Evergreen endpoint.
  • The algorithm for determining how to split up a resmoke.py test suite should prefer to generate more resmoke.py YAML suite files that are well under the target execution time than to generate any resmoke.py YAML suite files that are above the target execution time given the historical average of each test's execution time.

The Python script should create multiple resmoke.py YAML suite files in the buildscripts/resmokeconfig/suites/ directory along the lines of the following, with perhaps some additional information written to stdout describing what build variants the new Evergreen tasks should be run against.

sharding_000.yml

# This file was generated by buildscripts/generate_resmoke_suites.py and manual edits should also be
# made to sharding.yml in order to ensure running the full test suite locally matches the behavior
# of running parts of the test suite in Evergreen. It was generated against commit ...
# with the following options: ...
test_kind: js_test
 
selector:
  # The following tests take approximately
  #   17 minutes to run on Enterprise RHEL 6.2
  #   23 minutes to run on Linux DEBUG
  #   34 minutes to run on Enterprise Windows 2008R2
  roots:
  - jstests/sharding/aggregation_currentop.js
  - jstests/sharding/cleanup_orphaned_auth.js
  - jstests/sharding/crash_mongos_against_upgraded_cluster.js
  - jstests/sharding/cursor_valid_after_shard_stepdown.js
  - jstests/sharding/delete_during_migrate.js
  - jstests/sharding/mapReduce_nonSharded.js
  - jstests/sharding/migration_id_index.js
  - jstests/sharding/migration_ignore_interrupts_3.js
  - jstests/sharding/migration_server_status.js
  - jstests/sharding/min_optime_recovery_on_successful_move_chunk_commit.js
  - jstests/sharding/mongos_query_comment.js
  - jstests/sharding/mongos_shard_failure_tolerance.js
  - jstests/sharding/movechunk_with_default_paranoia.js
  - jstests/sharding/movechunk_with_moveParanoia.js
  - jstests/sharding/mr_shard_version.js
  - jstests/sharding/read_does_not_create_namespaces.js
  - jstests/sharding/shard_aware_init.js
  - jstests/sharding/shard_kill_and_pooling.js
  - jstests/sharding/tag_auto_split.js
  - jstests/sharding/tag_range.js
 
executor:
  config:
    shell_options:
      nodb: ''
      readMode: commands



 Comments   
Comment by Githook User [ 16/Apr/18 ]

Author:

{'name': 'David Bradford', 'email': 'david.bradford@mongodb.com', 'username': 'dbradf'}

Message: SERVER-34296: Scripts to breakup test suites into short running tasks
Branch: master
https://github.com/mongodb/mongo/commit/22ab3dc578d7272a627809314839a760681ba4f4

Generated at Thu Feb 08 04:36:11 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.