[SERVER-27408] burn_in_tests.py doesn't always exclude tagged tests when generating outfile Created: 13/Dec/16  Updated: 24/Feb/18  Resolved: 27/Jan/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.2.20, 3.4.5, 3.5.3

Type: Bug Priority: Major - P3
Reporter: Kyle Suarez Assignee: Robert Guo (Inactive)
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Related
Backwards Compatibility: Fully Compatible
Operating System: ALL
Backport Completed:
Backport Requested:
v3.4, v3.2
Sprint: TIG 2017-01-02, TIG 2017-02-13
Participants:
Linked BF Score: 0

 Description   

When burn_in_tests.py generates the list of executors and tests it will run, it doesn't always take tags properly into account. This can cause tests to be run in an inappropriate executor environment, which leads to spurious failures in CI.

I'm not exactly sure where the error is, but I assume it's somewhere in resmokelib.parser.create_test_membership_map().



 Comments   
Comment by Githook User [ 24/Feb/18 ]

Author:

{'email': 'robert.guo@10gen.com', 'name': 'Robert Guo', 'username': 'guoyr'}

Message: SERVER-27408 Allow resmoke to take more than one tag-related arguments.

Each tag argument can also be specified more than once and in both the
YAML suite definition and on the command line.

Tag argments will be agnostic to ordering, with each argument narrowing
down test files to be run. No tag is allowed to override another tag.

(cherry picked from commit a1363d00d04f0d235a88aad538d029448efac656)
(cherry picked from commit 2ba94da846a76ca4c2622c3bcb2837c2c128f080)
Branch: v3.2
https://github.com/mongodb/mongo/commit/896885c9e43d95b1796a45815937c3edd6b22065

Comment by Githook User [ 25/Apr/17 ]

Author:

{u'username': u'guoyr', u'name': u'Robert Guo', u'email': u'robert.guo@10gen.com'}

Message: SERVER-27408 Allow resmoke to take more than one tag-related arguments.

Each tag argument can also be specified more than once and in both the
YAML suite definition and on the command line.

Tag argments will be agnostic to ordering, with each argument narrowing
down test files to be run. No tag is allowed to override another tag.

(cherry picked from commit a1363d00d04f0d235a88aad538d029448efac656)
(cherry picked from commit 2ba94da846a76ca4c2622c3bcb2837c2c128f080)
Branch: v3.4
https://github.com/mongodb/mongo/commit/996be387c1c4caf598f6f8e38f44775a45219fe3

Comment by Githook User [ 27/Jan/17 ]

Author:

{u'username': u'guoyr', u'name': u'Robert Guo', u'email': u'robert.guo@10gen.com'}

Message: SERVER-27408 Fix evergreen configuration for arm64
Branch: master
https://github.com/mongodb/mongo/commit/2ba94da846a76ca4c2622c3bcb2837c2c128f080

Comment by Githook User [ 26/Jan/17 ]

Author:

{u'username': u'guoyr', u'name': u'Robert Guo', u'email': u'robert.guo@10gen.com'}

Message: SERVER-27408 Allow resmoke to take more than one tag-related arguments.

Each tag argument can also be specified more than once and in both the
YAML suite definition and on the command line.

Tag argments will be agnostic to ordering, with each argument narrowing
down test files to be run. No tag is allowed to override another tag.
Branch: master
https://github.com/mongodb/mongo/commit/a1363d00d04f0d235a88aad538d029448efac656

Comment by Kyle Suarez [ 06/Jan/17 ]

Is using argparse too out-of-scope?

Comment by Robert Guo (Inactive) [ 05/Jan/17 ]

The underlying problem is that burn_in_tests.py doesn't know the complete set of arguments passed to resmoke through evergreen.yml; which can cause tests to run in a wrong or invalid configuration. In this case, burn_in_tests missed the excludeWithAnyTags argument which would have made resmoke not run certain tests.

To get the list of arguments to resmoke, burn_in_tests.py manually parses evergreen.yml to extract the value of resmoke_args in a task. It also inherits all resmoke arguments defined on the variant such as test_flags and variant_excluded_flags. Unfortunately it doesn't know about any resmoke arguments defined in a task that isn't in resmoke_args, e.g. task_excluded_tags

One possible fix is to have burn_in_tests.py go through all the options in a task definition and manually handle them on a one-off basis. This seems brittle as options could have changed by the time they are passed to resmoke. (e.g. task_excluded_tags would be merged with variant_excluded_flags into the same resmoke option). burn_in_tests.py would need to repeat any logic in evergreen.yml that modifies these options.

The other possible fix is to make resmoke work better with arguments specified multiple times. E.g. it could allow multiple --excludeWithAnyTags flags and merge them internally so that task_excluded_tags and variant_excluded_flags won't need special handling. We can optionally have burn_in_tests check that the schema of a task definition does not change to prevent subtle breakages.

I'm leaning towards the 2nd option, even though it works best with argparse introduced in Python 2.7. Currently we use the older version called optparse, which does not allow multiple values for each keyword argument.

Comment by Kyle Suarez [ 13/Dec/16 ]

max.hirschhorn thought that this might be because there's a difference in burn_in_tests.py output depending on if changes in a repository are staged versus committed. But I ran a local test by doing

$ git apply work.patch
$ git add .
$ python buildscripts/burn_in_tests.py --buildVariant enterprise-rhel-62-64-bit --testListOutfile changes-staged-out.json --noExec
$ git commit -m "webscale"
$ python buildscripts/burn_in_tests.py --buildVariant enterprise-rhel-62-64-bit --testListOutfile changes-committed-out.json --noExec

Unfortunately I found no difference in the two outfiles.

Generated at Thu Feb 08 04:15:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.