[SERVER-51883] Provide mechanism for only running tests of flag-guarded features in variants where they are enabled Created: 29/Oct/20 Updated: 29/Oct/23 Resolved: 30/Mar/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Upgrade/Downgrade |
| Affects Version/s: | None |
| Fix Version/s: | 5.0.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Tess Avitabile (Inactive) | Assignee: | Robert Guo (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||
| Backport Requested: |
v4.9
|
||||||||||||||||||||||||||||||||||||
| Sprint: | Execution Team 2020-11-30, STM 2021-01-11, STM 2021-01-25, STM 2021-02-08, STM 2021-03-08, STM 2021-03-22, STM 2021-04-05 | ||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||||||||||||||||||||||
| Story Points: | 2 | ||||||||||||||||||||||||||||||||||||
| Description |
|
The feature flag-guarding policy states that feature flags should be enabled in the ~ Shared Library Enterprise RHEL 6.2 (disabled feature flags) variant and that tests for flag-guarded features should be tagged with the feature flag name. However, we don't have a mechanism for only running tests for flag-guarded features in the disabled feature flags variant. Current options include excluding those tags from all other variants (which requires too much copy-paste) or creating separate suites that only run in the disabled feature flags variant (which doesn't play well with passthroughs). Additionally, both of these solutions require significant changes to evergreen.yml when the feature flag is enabled. We should provide an easy and standard way for making sure that tests for flag-guarded features only run in variants in which they are enabled. |
| Comments |
| Comment by Githook User [ 30/Mar/21 ] | ||||||||||||||||||||||||||||||||||
|
Author: {'name': 'Robert Guo', 'email': 'robert.guo@mongodb.com'}Message: | ||||||||||||||||||||||||||||||||||
| Comment by Tess Avitabile (Inactive) [ 29/Mar/21 ] | ||||||||||||||||||||||||||||||||||
|
Great, thank you! The -runAllFeatureFlagTests arg sounds really useful. I would suggest updating the wiki and emailing the server contributors about this change. One question: Was -runAllFeatureFlagTests included in the sys-perf all feature flag builders as well? | ||||||||||||||||||||||||||||||||||
| Comment by Robert Guo (Inactive) [ 29/Mar/21 ] | ||||||||||||||||||||||||||||||||||
|
Here're a summary of the changes so far. It does not disable any tests at the moment. I will submit a followup commit for it.
| ||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 29/Mar/21 ] | ||||||||||||||||||||||||||||||||||
|
Author: {'name': 'Robert Guo', 'email': 'robert.guo@mongodb.com'}Message: | ||||||||||||||||||||||||||||||||||
| Comment by Tess Avitabile (Inactive) [ 29/Mar/21 ] | ||||||||||||||||||||||||||||||||||
|
Hi robert.guo! Would you mind explaining what the commit does? (Based on the commit name, I assume it sets all feature flags to true in the all feature flags builders, but I want to check.) Does it include any functionality to only run tests of flag-guarded features in variants where they are enabled? I'm asking in order to see whether this is change should be described to the server contributors and/or included in the wiki. | ||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 27/Mar/21 ] | ||||||||||||||||||||||||||||||||||
|
Author: {'name': 'Robert Guo', 'email': 'robert.guo@mongodb.com'}Message: | ||||||||||||||||||||||||||||||||||
| Comment by Brooke Miller [ 15/Dec/20 ] | ||||||||||||||||||||||||||||||||||
|
robert.guo will be writing up a mini-design ( | ||||||||||||||||||||||||||||||||||
| Comment by Benety Goh [ 02/Dec/20 ] | ||||||||||||||||||||||||||||||||||
|
| ||||||||||||||||||||||||||||||||||
| Comment by Benety Goh [ 02/Dec/20 ] | ||||||||||||||||||||||||||||||||||
|
To implement this filtering mechanism in the CI system, one possible strategy might be to:
| ||||||||||||||||||||||||||||||||||
| Comment by Benety Goh [ 02/Dec/20 ] | ||||||||||||||||||||||||||||||||||
|
Using featureFlagToaster as an example of an unreleased feature, we'd assume the following test should only be run on the disabled feature flags variant which includes the feature flag in its configuration.
While the default setting of the feature flag is set to false, we should not see this test run in any other build variant (eg. ! Shared Library Linux DEBUG). Without the filtering mechanism proposed in this ticket, a test would have to query a running server for the current state of the feature flag as follows:
One downside to the skipping the test using this runtime approach is that the test name would show up in the list of tests run in the builder, adding to the visual clutter that we would like to avoid using the declarative method. |