[SERVER-69883] Investigate improving JS test tags for tests dependent on in-development or recently-released feature work Created: 21/Sep/22  Updated: 21/Feb/23

Status: Open
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Engineering Test Priority: Major - P3
Reporter: Huayu Ouyang Assignee: Backlog - Replication Team
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-67268 Investigate simplifying usages of fea... Closed
is duplicated by SERVER-70203 Investigate using IDL to exclude JS t... Closed
Related
related to SERVER-67268 Investigate simplifying usages of fea... Closed
is related to SERVER-71467 Dont run feature flag specific tests ... Closed
is related to SERVER-70203 Investigate using IDL to exclude JS t... Closed
Assigned Teams:
Replication
Participants:

 Description   

Currently, in our test tagging system, there are various usages of featureFlagXX, requires_fcv_yy, and FeatureFlagUtil.isEnabled. (The current guidance for feature flag test tagging is here).

For example, currently if we have a JS test dependent on a feature flag, we tag it with featureFlagXX.
However, featureFlagXX tags aren't considered by the jstests/parallel/basic{,Plus}.js tests (only requires_fcv_yy would be) and featureFlagXX tags aren't considered by the jstests/core/selinux.js test (only no_selinux would be).

Additionally, for suites like replica_sets_multiversion, if they are being run on a feature flags enabled variant like Shared Library Enterprise RHEL 8.0 (all feature flags), even if the feature flag hasn’t been enabled by default and released yet, the feature flag will be enabled on this variant. So even if the test is tagged with featureFlagXX, since the feature flag is enabled, it will be run in replica_sets_multiversion, so we need to additionally tag it with requires_fcv_yy to prevent it from being run. Edit: As part of SERVER-71467, we determined that we should not be running multiversion tests with feature flags except for explicit multiversion tests. So, tagging tests with featureFlagXX should be sufficient for exclusion from suites like replica_sets_multiversion.

We should investigate if we can improve and simplify the tagging system so that featureFlagXX is both the exhaustive and definitive way to tag tests dependent on in-development or recently-released feature work.

Another solution is that the test exclusion code could consult the IDL to determine the FCV a given feature is enabled for, and exclude it appropriately, rather than needing to have both the feature flag tag and the fcv tag.

Additionally, we should investigate moving away from using FeatureFlagUtil.isEnabled where possible, since it is wasteful to start up a whole cluster only to see that the feature flag is not enabled and then skip the whole test.

As part of this investigation we can create a table which describes the different combinations of test of {in-development feature, test of feature in last-stable or earlier, test of feature in last-continuous but not last-stable} x {binVersion latest cluster, mixed binVersion last-stable + latest cluster, mixed binVersion last-continuous + latest cluster, pseudo-mixed latest cluster with future git tag cluster} for whether a test should or shouldn't run. And then we can map these back into whatever the build variants and implicit _multiversion test suites are or are not doing today in order to audit our testing coverage.



 Comments   
Comment by Huayu Ouyang [ 09/Feb/23 ]

This ticket should look into changes made in PM-3104 to see if there is any changes to the work identified for the ticket. Some potentially relevant changes are that PM-3104 will add a test exclusion readme with updated test tagging guidance (SERVER-72927), and a multiversion linter (SERVER-72930)

Generated at Thu Feb 08 06:14:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.