The filter_jstests() function in selector.py should be extended to optionally take at most one of include_tags or exclude_tags. The value associated with the include_tags or exclude_tags key should be an object containing a single key $allOf, $anyOf, or $not.

• If the key is $allOf or $anyOf, then its value should be a list of either (a) objects of same structure as described for the include_tags and exclude_tags keys, or (b) strings representing a literal tag value (a terminal value).
• If the key is $not, then its value should be either (a) an object of the same structure as described for the include_tags and exclude_tags keys, or (b) a string representing a literal tag value (a terminal value).

An example resmoke.py YAML suite configuration for expressing only running the reliable, non-resource intensive tests can be seen below.

selector: 
  roots: 
  - jstests/core/**/*.js
  exclude_tags: 
    $allOf: 
    - $anyOf: 
      - unreliable
      - unreliable|{task_name}
      - unreliable|{task_name}|{variant_name}
      - unreliable|{task_name}|{variant_name}|{distro_id}
    - resource_intensive

While only at most one of include_tags and exclude_tags can be specified in the YAML suite configuration, it should be possible to specifying none, either, or both of include_with_any_tags and exclude_with_any_tags at the same time in order to preserve support for the --includeWithAnyTags and --excludeWithAnyTags command line options.

• Specifying both include_tags and include_with_any_tags should effectively join them together with $allOf: [<include_tags>, <include_with_any_tags>].
• Specifying both include_tags and exclude_with_any_tags should effectively join them together with $allOf: [<include_tags>, {$not: <exclude_with_any_tags>}].
• Specifying both exclude_tags and exclude_with_any_tags should effectively join them together with $anyOf: [<exclude_tags>, <exclude_with_any_tags>].
• Specifying both exclude_tags and include_with_any_tags should effectively join them together with $anyOf: [<exclude_tags>, {$not: <include_with_any_tags>}].

Background

resmoke.py currently supports the command line options --includeWithAnyTags and --excludeWithAnyTags as well as the YAML suite configuration options include_with_any_tags and exclude_with_any_tags. The behavior of these options was changed in SERVER-27408 (along with removing the "all tags" variations in SERVER-27770) to make it so that specifying both an inclusion and exclusion would take the intersection of the included tests and whatever tests remain after apply the exclusion. This is sufficient for being able to run --excludeWithAnyTags=requires_journaling --includeWithAnyTags=unreliable and --excludeWithAnyTags=requires_journaling --excludeWithAnyTags=unreliable; however, it doesn't scale well when we would go to add further classifications of tests other than "reliable" and "unreliable". Additionally, it requires that no test suite or Evergreen task ever specifies --includeWithAnyTags.

The end goal of this work in combination with work in other tickets to is create a buildscripts/evergreen_run_tests.py script that replaces the contents of the "run tests" shell script in etc/evergreen.yml with logic that calls the main() in resmoke.py multiple times and sets different tag inclusions and exclusions to cover all combinations.