[SERVER-27408] burn_in_tests.py doesn't always exclude tagged tests when generating outfile Created: 13/Dec/16 Updated: 24/Feb/18 Resolved: 27/Jan/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 3.2.20, 3.4.5, 3.5.3 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Kyle Suarez | Assignee: | Robert Guo (Inactive) |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Operating System: | ALL | ||||||||||||
| Backport Completed: | |||||||||||||
| Backport Requested: |
v3.4, v3.2
|
||||||||||||
| Sprint: | TIG 2017-01-02, TIG 2017-02-13 | ||||||||||||
| Participants: | |||||||||||||
| Linked BF Score: | 0 | ||||||||||||
| Description |
|
When burn_in_tests.py generates the list of executors and tests it will run, it doesn't always take tags properly into account. This can cause tests to be run in an inappropriate executor environment, which leads to spurious failures in CI. I'm not exactly sure where the error is, but I assume it's somewhere in resmokelib.parser.create_test_membership_map(). |
| Comments |
| Comment by Githook User [ 24/Feb/18 ] | |||||
|
Author: {'email': 'robert.guo@10gen.com', 'name': 'Robert Guo', 'username': 'guoyr'}Message: Each tag argument can also be specified more than once and in both the Tag argments will be agnostic to ordering, with each argument narrowing (cherry picked from commit a1363d00d04f0d235a88aad538d029448efac656) | |||||
| Comment by Githook User [ 25/Apr/17 ] | |||||
|
Author: {u'username': u'guoyr', u'name': u'Robert Guo', u'email': u'robert.guo@10gen.com'}Message: Each tag argument can also be specified more than once and in both the Tag argments will be agnostic to ordering, with each argument narrowing (cherry picked from commit a1363d00d04f0d235a88aad538d029448efac656) | |||||
| Comment by Githook User [ 27/Jan/17 ] | |||||
|
Author: {u'username': u'guoyr', u'name': u'Robert Guo', u'email': u'robert.guo@10gen.com'}Message: | |||||
| Comment by Githook User [ 26/Jan/17 ] | |||||
|
Author: {u'username': u'guoyr', u'name': u'Robert Guo', u'email': u'robert.guo@10gen.com'}Message: Each tag argument can also be specified more than once and in both the Tag argments will be agnostic to ordering, with each argument narrowing | |||||
| Comment by Kyle Suarez [ 06/Jan/17 ] | |||||
|
Is using argparse too out-of-scope? | |||||
| Comment by Robert Guo (Inactive) [ 05/Jan/17 ] | |||||
|
The underlying problem is that burn_in_tests.py doesn't know the complete set of arguments passed to resmoke through evergreen.yml; which can cause tests to run in a wrong or invalid configuration. In this case, burn_in_tests missed the excludeWithAnyTags argument which would have made resmoke not run certain tests. To get the list of arguments to resmoke, burn_in_tests.py manually parses evergreen.yml to extract the value of resmoke_args in a task. It also inherits all resmoke arguments defined on the variant such as test_flags and variant_excluded_flags. Unfortunately it doesn't know about any resmoke arguments defined in a task that isn't in resmoke_args, e.g. task_excluded_tags One possible fix is to have burn_in_tests.py go through all the options in a task definition and manually handle them on a one-off basis. This seems brittle as options could have changed by the time they are passed to resmoke. (e.g. task_excluded_tags would be merged with variant_excluded_flags into the same resmoke option). burn_in_tests.py would need to repeat any logic in evergreen.yml that modifies these options. The other possible fix is to make resmoke work better with arguments specified multiple times. E.g. it could allow multiple --excludeWithAnyTags flags and merge them internally so that task_excluded_tags and variant_excluded_flags won't need special handling. We can optionally have burn_in_tests check that the schema of a task definition does not change to prevent subtle breakages. I'm leaning towards the 2nd option, even though it works best with argparse introduced in Python 2.7. Currently we use the older version called optparse, which does not allow multiple values for each keyword argument. | |||||
| Comment by Kyle Suarez [ 13/Dec/16 ] | |||||
|
max.hirschhorn thought that this might be because there's a difference in burn_in_tests.py output depending on if changes in a repository are staged versus committed. But I ran a local test by doing
Unfortunately I found no difference in the two outfiles. |