[SERVER-51883] Provide mechanism for only running tests of flag-guarded features in variants where they are enabled Created: 29/Oct/20  Updated: 29/Oct/23  Resolved: 30/Mar/21

Status: Closed
Project: Core Server
Component/s: Upgrade/Downgrade
Affects Version/s: None
Fix Version/s: 5.0.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Tess Avitabile (Inactive) Assignee: Robert Guo (Inactive)
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Issue split
split to SERVER-53367 Mini-Design to enable feature flag te... Closed
Problem/Incident
Related
is related to SERVER-56383 Feature flag tag is not preventing a ... Closed
is related to SERVER-51112 Make helper for targeted upgrade/down... Closed
is related to SERVER-51877 Check that all API Version 1 commands... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v4.9
Sprint: Execution Team 2020-11-30, STM 2021-01-11, STM 2021-01-25, STM 2021-02-08, STM 2021-03-08, STM 2021-03-22, STM 2021-04-05
Participants:
Linked BF Score: 0
Story Points: 2

 Description   

The feature flag-guarding policy states that feature flags should be enabled in the ~ Shared Library Enterprise RHEL 6.2 (disabled feature flags) variant and that tests for flag-guarded features should be tagged with the feature flag name. However, we don't have a mechanism for only running tests for flag-guarded features in the disabled feature flags variant. Current options include excluding those tags from all other variants (which requires too much copy-paste) or creating separate suites that only run in the disabled feature flags variant (which doesn't play well with passthroughs). Additionally, both of these solutions require significant changes to evergreen.yml when the feature flag is enabled.

We should provide an easy and standard way for making sure that tests for flag-guarded features only run in variants in which they are enabled.



 Comments   
Comment by Githook User [ 30/Mar/21 ]

Author:

{'name': 'Robert Guo', 'email': 'robert.guo@mongodb.com'}

Message: SERVER-51883 don't run feature flag tests if --runAllFeatureFlagTests arg is not specified
Branch: master
https://github.com/mongodb/mongo/commit/d9761413b91ad7428fc7500ed01ef71bca190e6a

Comment by Tess Avitabile (Inactive) [ 29/Mar/21 ]

Great, thank you! The -runAllFeatureFlagTests arg sounds really useful.

I would suggest updating the wiki and emailing the server contributors about this change.

One question: Was -runAllFeatureFlagTests included in the sys-perf all feature flag builders as well?

Comment by Robert Guo (Inactive) [ 29/Mar/21 ]

Here're a summary of the changes so far. It does not disable any tests at the moment. I will submit a followup commit for it.

  • All feature flags that exist will now be enabled by default on mongos and mongod when running resmoke with the -runAllFeatureFlagTests arg. This arg is used on the all feature flag builders.
  • There's a new standalone config file, fully_disabled_feature_flags.yml, that contains a list of feature flags to be disabled by default on the all feature flags builder. This file is referenced in a comment in all these builders for self-service disabling of feature flags. 
  • Fully disabled feature flags can be overridden on a per-task or per-test basis by passing in the relevant --setParameter featureFlagXYZ arguments to the test fixtures. Here's an example that does it through resmoke.py's commandline args in an Evergreen task definition.
Comment by Githook User [ 29/Mar/21 ]

Author:

{'name': 'Robert Guo', 'email': 'robert.guo@mongodb.com'}

Message: SERVER-51883 disable benchmark tests on feature flag variants
Branch: master
https://github.com/mongodb/mongo/commit/83fdf9063cfd3a69aa1c3dc2b9566142dc51d634

Comment by Tess Avitabile (Inactive) [ 29/Mar/21 ]

Hi robert.guo! Would you mind explaining what the commit does? (Based on the commit name, I assume it sets all feature flags to true in the all feature flags builders, but I want to check.) Does it include any functionality to only run tests of flag-guarded features in variants where they are enabled? I'm asking in order to see whether this is change should be described to the server contributors and/or included in the wiki.

Comment by Githook User [ 27/Mar/21 ]

Author:

{'name': 'Robert Guo', 'email': 'robert.guo@mongodb.com'}

Message: SERVER-51883 Add all feature flags by default to the all feature flags
builders
Branch: master
https://github.com/mongodb/mongo/commit/8e8af7bb449660a932cfbf5a9dc4101ef7fa2eea

Comment by Brooke Miller [ 15/Dec/20 ]

robert.guo will be writing up a mini-design (SERVER-53367) before implementing this. Once we have the path forward, we'll schedule this ticket to a sprint.

Comment by Benety Goh [ 02/Dec/20 ]

SERVER-51877 provides a template for a script that scans the IDL files in the source tree.

Comment by Benety Goh [ 02/Dec/20 ]

To implement this filtering mechanism in the CI system, one possible strategy might be to:

  • Extend the build system to process the IDL files containing feature flags to create a list of existing feature flags and their current settings.
    • This could also be done in a standalone script or program.
  • The CI system would apply, for the builder it is running, any feature flag --setParameter settings to the list generated by the build system.
  • The CI system would add to the resmoke.py invocation additional --excludeWithAnyTags options to ensure that tests for unreleased features would be skipped.
Comment by Benety Goh [ 02/Dec/20 ]

Using featureFlagToaster as an example of an unreleased feature, we'd assume the following test should only be run on the disabled feature flags variant which includes the feature flag in its configuration.

toaster.js

/**
 * New test for toaster feature.
 * @tags: [
 *     featureFlagToaster
 * ]
 */
 
(function() {
    'use strict';
 
   // No need to check runtime settings on the server containing 'db' - toaster feature should be enabled if this test is run.
  assert.commandWorked(db.runCommand({toaster: true}));
})();

While the default setting of the feature flag is set to false, we should not see this test run in any other build variant (eg. ! Shared Library Linux DEBUG).

Without the filtering mechanism proposed in this ticket, a test would have to query a running server for the current state of the feature flag as follows:

toaster.js

/**
 * New test for toaster feature.
 * @tags: [
 *     featureFlagToaster
 * ]
 */
 
(function() {
    'use strict';
 
    const isFeatureEnabled = db.adminCommand({getParameter: 1, featureFlagToaster: 1}).featureFlagToaster.value;
    if (!isFeatureEnabled) {
        jsTestLog('Skipping test because feature is not available.');
        return;
    }
 
   // No need to check runtime settings on the server containing 'db' - toaster feature should be enabled if this test is run.
  assert.commandWorked(db.runCommand({toaster: true}));
})();

One downside to the skipping the test using this runtime approach is that the test name would show up in the list of tests run in the builder, adding to the visual clutter that we would like to avoid using the declarative method.

Generated at Thu Feb 08 05:26:42 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.