[SERVER-39564] burn_in_tests.py doesn't run tests in the configuration for *_gen Evergreen tasks Created: 13/Feb/19  Updated: 29/Oct/23  Resolved: 14/Feb/19

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 4.0.7, 4.1.9

Type: Bug Priority: Critical - P2
Reporter: Max Hirschhorn Assignee: David Bradford (Inactive)
Resolution: Fixed Votes: 0
Labels: tig-burnin
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Requested:
v4.0
Steps To Reproduce:

I applied the following patch to track down where in the burn_in_tests.py script things were going wrong.

$ git checkout 8e5e745e98d33633e7d24a2629f22cdba79d9851
$ python buildscripts/resmoke.py --findSuites jstests/auth/getMore.js
[resmoke] 2019-02-12T23:51:37.998-0500 jstests/auth/getMore.js will be run by the following suite(s): ['auth', 'auth_audit']
$ python buildscripts/burn_in_tests.py --branch=master --buildVariant=enterprise-rhel-62-64-bit --testListOutfile=jstests/new_tests.json --noExec --checkEvergreen
Comparing current branch against 7a7baa2539ec169335086e45c7d0b85ba7cdb877
@@@ changed_tests ['jstests/auth/getMore.js']
@@@ tests_by_executor defaultdict(<type 'list'>, {'auth_audit': ['jstests/auth/getMore.js'], 'auth': ['jstests/auth/getMore.js']})
@@@ skipping task compile
...
@@@ skipping task auth_gen
@@@ skipping task auth_audit_gen
...
@@@ tests_by_task {}

diff --git a/buildscripts/burn_in_tests.py b/buildscripts/burn_in_tests.py
index cae3ee9bac..0bb0aae000 100644
--- a/buildscripts/burn_in_tests.py
+++ b/buildscripts/burn_in_tests.py
@@ -146,7 +146,7 @@ def find_changed_tests(branch_name, base_commit, max_revisions, buildvariant, ch
         # activated.
         revs_to_check = callo(["git", "rev-list", base_commit, "--max-count=200",
                                "--skip=1"]).splitlines()
-        last_activated = find_last_activated_task(revs_to_check, buildvariant, branch_name)
+        last_activated = "7a7baa2539ec169335086e45c7d0b85ba7cdb877"
         if last_activated is None:
             # When the current commit is the first time 'buildvariant' has run, there won't be a
             # commit among 'revs_to_check' that's been activated in Evergreen. We handle this by
@@ -282,6 +282,8 @@ def create_task_list(evergreen_conf, buildvariant, suites, exclude_tasks):
             resmoke_args = task.combined_resmoke_args
             if resmoke_args:
                 variant_task_args[task.name] = resmoke_args
+            else:
+                print '@@@ skipping task', task.name
 
     # Create the list of tasks to run for the specified suite.
     tasks_to_run = {}
@@ -363,6 +365,7 @@ def main():
                                            values.buildvariant, values.check_evergreen)
         exclude_suites, exclude_tasks, exclude_tests = find_exclude_tests(values.selector_file)
         changed_tests = filter_tests(changed_tests, exclude_tests)
+        print '@@@ changed_tests', changed_tests
         # If there are no changed tests, exit cleanly.
         if not changed_tests:
             print "No new or modified tests found."
@@ -372,8 +375,10 @@ def main():
         suites = resmokelib.suitesconfig.get_suites(
             suite_files=values.suite_files.split(","), test_files=changed_tests)
         tests_by_executor = create_executor_list(suites, exclude_suites)
+        print '@@@ tests_by_executor', tests_by_executor
         tests_by_task = create_task_list(evergreen_conf, values.buildvariant, tests_by_executor,
                                          exclude_tasks)
+        print '@@@ tests_by_task', tests_by_task
         if values.test_list_outfile is not None:
             _write_report_file(tests_by_task, values.test_list_outfile)

Sprint: DAG 2019-02-25
Participants:

 Description   

I noticed this upon reviewing my patch build for backporting SERVER-35551 to the 4.0 branch. I then took a look at the master branch and found this task had run tests via burn_in_tests.py but this other task didn't run tests via burn_in_tests.py despite the jstests/auth/getMore.js test being modified.

The issue stems from how the task definitions for the *_gen Evergreen tasks have a different style compared to before and leads to task.combined_resmoke_args being None and for them to be skipped over in the tests_by_task mapping.

- name: auth_gen
  commands:
  - func: "generate resmoke tasks"
    vars:
      task: auth
      resmoke_args: --storageEngine=wiredTiger
      fallback_num_sub_suites: 4

- <<: *task_template
  name: auth
  commands:
  - func: "do setup"
  - func: "run tests"
    vars:
      resmoke_args: --suites=auth --storageEngine=wiredTiger



 Comments   
Comment by Githook User [ 14/Feb/19 ]

Author:

{'name': 'David Bradford', 'email': 'david.bradford@mongodb.com', 'username': 'dbradf'}

Message: SERVER-39564: Fix burn_in_tests for _gen tasks

(cherry picked from commit 8fac3580ab40811cbcb078ec9e1e2f16bfbf23a1)
Branch: v4.0
https://github.com/mongodb/mongo/commit/744ea7109573bb19768559dee8132f0cf818b8b4

Comment by Githook User [ 14/Feb/19 ]

Author:

{'name': 'David Bradford', 'email': 'david.bradford@mongodb.com', 'username': 'dbradf'}

Message: SERVER-39564: Fix burn_in_tests for _gen tasks
Branch: master
https://github.com/mongodb/mongo/commit/8fac3580ab40811cbcb078ec9e1e2f16bfbf23a1

Generated at Thu Feb 08 04:52:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.