[SERVER-30980] Add support for more complex tag matching in resmoke.py Created: 07/Sep/17  Updated: 30/Oct/23  Resolved: 27/Sep/17

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 3.6.0-rc0

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: Yves Duhem
Resolution: Fixed Votes: 0
Labels: tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
causes SERVER-35190 resmoke.py runs lists of tests in alp... Closed
Related
related to SERVER-31309 Unit test files cannot be specified a... Closed
related to SERVER-31312 resmoke changed its interpretation of... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-10-02
Participants:

 Description   

The filter_jstests() function in selector.py should be extended to optionally take at most one of include_tags or exclude_tags. The value associated with the include_tags or exclude_tags key should be an object containing a single key $allOf, $anyOf, or $not.

  • If the key is $allOf or $anyOf, then its value should be a list of either (a) objects of same structure as described for the include_tags and exclude_tags keys, or (b) strings representing a literal tag value (a terminal value).
  • If the key is $not, then its value should be either (a) an object of the same structure as described for the include_tags and exclude_tags keys, or (b) a string representing a literal tag value (a terminal value).

An example resmoke.py YAML suite configuration for expressing only running the reliable, non-resource intensive tests can be seen below.

selector:
  roots:
  - jstests/core/**/*.js
  exclude_tags:
    $allOf:
    - $anyOf:
      - unreliable
      - unreliable|{task_name}
      - unreliable|{task_name}|{variant_name}
      - unreliable|{task_name}|{variant_name}|{distro_id}
    - resource_intensive

While only at most one of include_tags and exclude_tags can be specified in the YAML suite configuration, it should be possible to specifying none, either, or both of include_with_any_tags and exclude_with_any_tags at the same time in order to preserve support for the --includeWithAnyTags and --excludeWithAnyTags command line options.

  • Specifying both include_tags and include_with_any_tags should effectively join them together with $allOf: [<include_tags>, <include_with_any_tags>].
  • Specifying both include_tags and exclude_with_any_tags should effectively join them together with $allOf: [<include_tags>, {$not: <exclude_with_any_tags>}].
  • Specifying both exclude_tags and exclude_with_any_tags should effectively join them together with $anyOf: [<exclude_tags>, <exclude_with_any_tags>].
  • Specifying both exclude_tags and include_with_any_tags should effectively join them together with $anyOf: [<exclude_tags>, {$not: <include_with_any_tags>}].
Background

resmoke.py currently supports the command line options --includeWithAnyTags and --excludeWithAnyTags as well as the YAML suite configuration options include_with_any_tags and exclude_with_any_tags. The behavior of these options was changed in SERVER-27408 (along with removing the "all tags" variations in SERVER-27770) to make it so that specifying both an inclusion and exclusion would take the intersection of the included tests and whatever tests remain after apply the exclusion. This is sufficient for being able to run --excludeWithAnyTags=requires_journaling --includeWithAnyTags=unreliable and --excludeWithAnyTags=requires_journaling --excludeWithAnyTags=unreliable; however, it doesn't scale well when we would go to add further classifications of tests other than "reliable" and "unreliable". Additionally, it requires that no test suite or Evergreen task ever specifies --includeWithAnyTags.

The end goal of this work in combination with work in other tickets to is create a buildscripts/evergreen_run_tests.py script that replaces the contents of the "run tests" shell script in etc/evergreen.yml with logic that calls the main() in resmoke.py multiple times and sets different tag inclusions and exclusions to cover all combinations.



 Comments   
Comment by Githook User [ 27/Sep/17 ]

Author:

{'email': 'yves.duhem@mongodb.com', 'name': 'Yves Duhem', 'username': 'syev'}

Message: SERVER-30980 Add tag selector expressions support
Branch: master
https://github.com/mongodb/mongo/commit/02e1ac97d8453dfbf7b9ac2cb5a5ee3abf3ef45e

Comment by Max Hirschhorn [ 07/Sep/17 ]

yves.duhem, in my proof-of-concept for this idea, I had defined classes similar to what exists in the server's codebase for the MatchExpression language and parsed the $allOf, $anyOf, and $not objects into an instance of these classes. I think that it may be easier to test the parsing logic separately from the execution logic. Another aspect for this idea that I had played around with was making a Selector class that took an os_path_module parameter (that defaults to os.path) and similar parameters for the globstar and jscomment modules) in order to make it easier to mock out anything that needs to read from disk. We should discuss when you get back from vacation whether that approach is feasible or whether we'd want to test a subcomponent of the logic that receives the list of tags associated with a file.

class AllOfExpression(object):
 
    def __init__(self, children):
        self.__children = children
 
    def __call__(self, file_tags):
        return all(child(file_tags) for child in children)
 
 
class AnyOfExpression(object):
 
    def __init__(self, children):
        self.__children = children
 
    def __call__(self, file_tags):
        return any(child(file_tags) for child in children)
 
 
class NotExpression(object):
    def __init__(self, child):
        self.__child = child
 
    def __call__(self, file_tags):
        return not child(file_tags)

Generated at Thu Feb 08 04:25:38 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.