[SERVER-60684] Exclude Tests and Tasks tagged with "serverless" from running on last-lts Created: 14/Oct/21  Updated: 16/Oct/23

Status: Open
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Robert Guo (Inactive) Assignee: Backlog - Service Architecture
Resolution: Unresolved Votes: 0
Labels: branch-6.0
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
depends on SERVER-73786 Move serverless tests from jstests/re... Open
depends on SERVER-66970 Add EOL'd releases information to the... Closed
Related
related to SERVER-62637 Find a way to avoid running shardMerg... Closed
is related to SERVER-63596 Exclude last LTS from multiversion se... Open
is related to SERVER-66970 Add EOL'd releases information to the... Closed
is related to SERVER-73803 Update tags for tests that use featur... Closed
Assigned Teams:
Service Arch
Backport Requested:
v6.1, v6.0
Sprint: STM 2021-12-13, STM 2022-03-07, DAG 2022-06-13, Server Serverless 2022-12-26, Server Serverless 2023-01-09, Server Serverless 2023-01-23, Server Serverless 2023-02-06, Server Serverless 2023-02-20, Server Serverless 2023-05-01
Participants:
Linked BF Score: 35
Story Points: 1

 Description   

Build a mechanism to prevent Evergreen tasks and JS tests tagged with "serverless-only" from running on last-lts branches. The implementation should allow for easy expansion to enable arbitrary features to not run on arbitrary branches.

Impl. sketch:
1. Add an --excludeWithAnyTags here for JS test tags.
2. Add a should_skip() function to task_def here and have it check if the tag exists for the task.
3. Similar to 1, but in evergreen_burn_in_tests.py
4. Similar to 2, but in evergreen_burn_in_tests.py



 Comments   
Comment by Esha Maharishi (Inactive) [ 21/Dec/22 ]

We decided Server Serverless will manually delete the tests on 6.0.

Comment by Steven Vannelli [ 07/Dec/22 ]

Ah fantastic, thanks juan.gu@mongodb.com!

Comment by Juan Gu [ 07/Dec/22 ]

steven.vannelli@mongodb.com , It has been scheduled and should be fixed by end of this week or early next week at the latest.

Comment by Steven Vannelli [ 07/Dec/22 ]

alexander.neben@mongodb.com juan.gu@mongodb.com just checking in on this one since the associated hot BF (BF-26914) is still open. Do we have a plan for being able to close that out?

Comment by Alex Neben [ 28/Nov/22 ]

1. Create another ticket to replace this one for 6.0
2. Make the change
3. Get it onto release team's post release workflow

Comment by Esha Maharishi (Inactive) [ 09/Nov/22 ]

Bringing this back into Needs Scheduling for SDP:

Now that cloud is running serverless on v6.1, we'd like to stop running serverless tests on the v6.0 branch.

We had filed this ticket last year when we similarly wanted to stop running serverless tests on v5.0. As a stop-gap we manually deleted the serverless tests from v5.0.

I wanted to check in if SDP could automate this process, since otherwise we will have to take this manual step every year. We've tagged the tests that should stop being run with a "serverless" test tag (example).

Comment by Robert Guo (Inactive) [ 13/Apr/22 ]

(Update) This is a different request. Reopening.

Added a comment to the linked ticket and closing this as a dupe.

Comment by Githook User [ 31/Jan/22 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: SERVER-60684 Delete Serverless tests from 5.0 branch
Branch: v5.0
https://github.com/mongodb/mongo/commit/818f05bdcd259630a164e0ec5c068ed94dc943de

Comment by Githook User [ 28/Jan/22 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: SERVER-60684 Tag serverless tests, 2
Branch: master
https://github.com/mongodb/mongo/commit/7d739f33ae41f84a63e96be68326ea52b9547815

Comment by Githook User [ 28/Jan/22 ]

Author:

{'name': 'A. Jesse Jiryu Davis', 'email': 'jesse@mongodb.com', 'username': 'ajdavis'}

Message: SERVER-60684 Tag serverless tests
Branch: master
https://github.com/mongodb/mongo/commit/258ceaa9a9ba36fc6656c65f752bda39157a5029

Comment by A. Jesse Jiryu Davis [ 28/Jan/22 ]

We have enough info in multiversionconstants.py to determine if the current branch is latest, last-continuous, and/or last-lts.

Plan:

  • 5.1 testing in general is being disabled presently
  • On the 5.0 branch, delete all serverless test files and tasks (Jesse)
  • On master, tag all serverless tests & tasks with "serverless" (Jesse)
  • Update testing logic: if the current branch being tested is older than last-continuous, skip all tests & tasks tagged "serverless" (STM team)

The last rule only really matters for last-lts, since we quickly stop testing entirely on other branches older than last-continuous.

We could rename the "serverless" tag to something like "skip-if-older-than-last-continuous" if we find non-Serverless uses for it.

Comment by A. Jesse Jiryu Davis [ 27/Jan/22 ]

Esha and Louisa tell me we can disable serverless tests for 5.0 and 5.1.

Comment by Robert Guo (Inactive) [ 14/Dec/21 ]

Thinking about this a bit more, I realized we don't currently have a way for an lts branch to know that it's LTS. Doing so would likely require a manual change to the release branch, and may not be meaningfully better than manually removing the tests during branching using the steps described in this ticket description since it is only once a year. I'm going to put this ticket under the 6.0 branching work with a label and sync up with the build barons to collaborate on this change going forward.

Reducing redundant test coverage for continuous releases is being initially discussed in PROG-5936. Most of the work there may be on the cloud side; we would still have to manually enable or disable an Evergreen project.

Comment by Brooke Miller [ 19/Oct/21 ]

robert.guo is going to file a separate ticket in order to allow us to enforce tags for modified tests and suites

Comment by Robert Guo (Inactive) [ 16/Oct/21 ]

Yeah that would be nice. We can update the ticket to have tests be tagged based on features and add a new linter rule to check added/moved tests have one or more feature tag.

Comment by Judah Schvimer [ 14/Oct/21 ]

Should this be more general, to exclude serverless-only tests from any branch besides last-continuous?

Also, should we tag tests with features and tag features with serverless-only so we can remove serverless-only easily from features in the future if desirable?

Generated at Thu Feb 08 05:50:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.