- resmoke.py doesn't know the individual tests being run and therefore cannot apply tag-based exclusion using exclude_any_with_tags.
- resmoke.py doesn't spawn separate mongo shell processes for the individual tests being run and therefore cannot (a) create separate log endpoints for their output or (b) report separate pass/fail statuses.
This ticket is only intended to address #1. Addressing #2a is difficult due to the existing logkeeper schema because it makes an assumption that a test has ended as soon as another test that's part of the same build_id has started. Addressing #2b is difficult because creating a new test_id is tied in resmoke.py to starting a test.
A new parallel_js_test test kind should be introduced that makes use of resmoke.py's buildscripts/resmokelib/selector.py module within the ParallelJSTestCase class to filter out tests from the jstests/core/ directory that shouldn't be run by the jstests/parallel/basic*.js tests. ParallelJSTestCase._make_process() should spawn a mongo shell process with a new TestData.testSchedule array option (or similar name) where each element corresponds to the list of tests for a ScopedThread spawned by the jstests/parallel/basic*.js tests to run.
All of the logic of the ParallelTester.createJstestsLists() function should be expressed in the resmoke.py YAML suite file and performed by the ParallelJSTestCase class. In particular,
- These tests should be automatically excluded when resmoke.py is invoked with --excludeWithAnyTags=requires_find_command as the parallel_compatibility Evergreen task configures it.
- The order of all of the tests in TestData.testSchedule, including those mentioned in the executor.config.serial_execution section should be shuffled.
- It should be an error to explicitly mention a test in the executor.config.selector section that doesn't exist. (This should happen automatically.)
- It should be an error to explicitly mention a test in the executor.config.serial_execution section that doesn't exist.
A few other notes:
- The number of array elements to generate in TestData.testSchedule should be defined as a constant (=4) on the ParallelJSTestCase class but need not be configurable via YAML.
- New parallel_jscore_passthrough and parallel_jscore_compatibility_passthrough Evergreen tasks should be introduced to all build variants that currently run the parallel and parallel_compatibility tasks, respectively.