[SERVER-69883] Investigate improving JS test tags for tests dependent on in-development or recently-released feature work Created: 21/Sep/22 Updated: 21/Feb/23 |
|
| Status: | Open |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Engineering Test | Priority: | Major - P3 |
| Reporter: | Huayu Ouyang | Assignee: | Backlog - Replication Team |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Assigned Teams: |
Replication
|
||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Description |
|
Currently, in our test tagging system, there are various usages of featureFlagXX, requires_fcv_yy, and FeatureFlagUtil.isEnabled. (The current guidance for feature flag test tagging is here). For example, currently if we have a JS test dependent on a feature flag, we tag it with featureFlagXX. Additionally, for suites like replica_sets_multiversion, if they are being run on a feature flags enabled variant like Shared Library Enterprise RHEL 8.0 (all feature flags), even if the feature flag hasn’t been enabled by default and released yet, the feature flag will be enabled on this variant. So even if the test is tagged with featureFlagXX, since the feature flag is enabled, it will be run in replica_sets_multiversion, so we need to additionally tag it with requires_fcv_yy to prevent it from being run. Edit: As part of We should investigate if we can improve and simplify the tagging system so that featureFlagXX is both the exhaustive and definitive way to tag tests dependent on in-development or recently-released feature work. Another solution is that the test exclusion code could consult the IDL to determine the FCV a given feature is enabled for, and exclude it appropriately, rather than needing to have both the feature flag tag and the fcv tag. Additionally, we should investigate moving away from using FeatureFlagUtil.isEnabled where possible, since it is wasteful to start up a whole cluster only to see that the feature flag is not enabled and then skip the whole test. As part of this investigation we can create a table which describes the different combinations of test of {in-development feature, test of feature in last-stable or earlier, test of feature in last-continuous but not last-stable} x {binVersion latest cluster, mixed binVersion last-stable + latest cluster, mixed binVersion last-continuous + latest cluster, pseudo-mixed latest cluster with future git tag cluster} for whether a test should or shouldn't run. And then we can map these back into whatever the build variants and implicit _multiversion test suites are or are not doing today in order to audit our testing coverage. |
| Comments |
| Comment by Huayu Ouyang [ 09/Feb/23 ] |
|
This ticket should look into changes made in PM-3104 to see if there is any changes to the work identified for the ticket. Some potentially relevant changes are that PM-3104 will add a test exclusion readme with updated test tagging guidance ( |