[SERVER-39362] Make parallel/basic.js test respect resmoke.py tag-based exclusions Created: 04/Feb/19  Updated: 27/Oct/23  Resolved: 25/Aug/23

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Max Hirschhorn Assignee: [DO NOT ASSIGN] Backlog - Server Development Platform Team (SDP) (Inactive)
Resolution: Gone away Votes: 10
Labels: tig-qwin-eligible, tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-73485 Remove jstests/parallel/basic* Closed
Related
related to SERVER-80452 Complete TODO listed in SERVER-39362 Closed
related to SERVER-36256 Make "wiredTiger" default value for j... Backlog
is related to SERVER-62557 Parallel tester ignores feature flagging Closed
Assigned Teams:
Server Development Platform
Participants:
Linked BF Score: 70
Story Points: 4

 Description   

The jstests/parallel/basic*.js tests have a couple shortcomings due to how they are a JavaScript test that runs other JavaScript tests:

  1. resmoke.py doesn't know the individual tests being run and therefore cannot apply tag-based exclusion using exclude_any_with_tags.
  2. resmoke.py doesn't spawn separate mongo shell processes for the individual tests being run and therefore cannot (a) create separate log endpoints for their output or (b) report separate pass/fail statuses.

This ticket is only intended to address #1. Addressing #2a is difficult due to the existing logkeeper schema because it makes an assumption that a test has ended as soon as another test that's part of the same build_id has started. Addressing #2b is difficult because creating a new test_id is tied in resmoke.py to starting a test.

A new parallel_js_test test kind should be introduced that makes use of resmoke.py's buildscripts/resmokelib/selector.py module within the ParallelJSTestCase class to filter out tests from the jstests/core/ directory that shouldn't be run by the jstests/parallel/basic*.js tests. ParallelJSTestCase._make_process() should spawn a mongo shell process with a new TestData.testSchedule array option (or similar name) where each element corresponds to the list of tests for a ScopedThread spawned by the jstests/parallel/basic*.js tests to run.

TestData.testSchedule = [
  [
    // This list should be sourced exclusively from the executor.config.serial_execution section.
    '0',
    'jstests/core/killop_drop_collection.js',
    'jstests/core/fsync.js',
    'jstests/core/currentop.js',
    ...
  ],
  [
    // This list and the following ones should be sourced exclusively from tests filted by the
    // executor.config.selector section that aren't present in the executor.config.serial_execution
    // section.
    '1',
    'jstests/core/all.js',
    ...
  ],
  [
    '2',
    'jstests/core/all2.js',
    ...
  ],
  [
    '3',
    'jstests/core/all3.js',
    ...
  ],
];

All of the logic of the ParallelTester.createJstestsLists() function should be expressed in the resmoke.py YAML suite file and performed by the ParallelJSTestCase class. In particular,

  • These tests should be automatically excluded when resmoke.py is invoked with --excludeWithAnyTags=requires_find_command as the parallel_compatibility Evergreen task configures it.
  • The order of all of the tests in TestData.testSchedule, including those mentioned in the executor.config.serial_execution section should be shuffled.
  • It should be an error to explicitly mention a test in the executor.config.selector section that doesn't exist. (This should happen automatically.)
  • It should be an error to explicitly mention a test in the executor.config.serial_execution section that doesn't exist.

A few other notes:

  • The number of array elements to generate in TestData.testSchedule should be defined as a constant (=4) on the ParallelJSTestCase class but need not be configurable via YAML.
  • New parallel_jscore_passthrough and parallel_jscore_compatibility_passthrough Evergreen tasks should be introduced to all build variants that currently run the parallel and parallel_compatibility tasks, respectively.

buildscripts/resmokeconfig/suites/parallel_jscore_passthrough.yml

test_kind: parallel_js_test
 
selector:
  roots:
  - jstests/parallel/basic*.js
  
executor:
  archive:
    hooks:
    - ValidateCollections
  config:
    selector:
      roots:
      - jstests/core/**/*.js
      exclude_files:
      # Transactions are not supported on MongoDB standalone nodes.
      - jstests/core/txns/**/*.js
 
    serial_execution:
    # The following tests run the {fsync: 1, lock: 1} command.
    - jstests/core/currentop.js
    - jstests/core/fsync.js
    - jstests/core/killop_drop_collection.js
    ...
 
    shell_options:
      readMode: commands
  hooks:
  - class: ValidateCollections
  fixture:
    class: MongoDFixture
    mongod_options:
      set_parameters:
        enableTestCommands: 1



 Comments   
Comment by Dan Larkin-York [ 25/Aug/23 ]

The parallel/basic* tests were removed by SERVER-73485. Closing as Gone Away.

Comment by Connie Chen [ 03/Apr/23 ]

This will go away once SERVER-73485 is complete

Comment by Max Hirschhorn [ 09/Jan/23 ]

Is this ticket thought to be the "rewrite parallel suite to be run by resmoke.py" ticket? Or is there some other ticket tracking that?

This is that ticket.

Comment by Charlie Swanson [ 09/Jan/23 ]

Is this ticket thought to be the "rewrite parallel suite to be run by resmoke.py" ticket? Or is there some other ticket tracking that?

Comment by James Wahlin [ 27/Feb/19 ]

+1 for this feature. It is surprising that tags are not respected by the parallel suites and non-intuitive that a separate blacklist exists and requires updating.

Generated at Thu Feb 08 04:51:48 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.