[SERVER-62557] Parallel tester ignores feature flagging Created: 12/Jan/22 Updated: 09/Jan/23 Resolved: 16/Feb/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Gregory Noma | Assignee: | Zituo Jin |
| Resolution: | Won't Fix | Votes: | 0 |
| Labels: | faster-patches | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Operating System: | ALL | ||||||||||||
| Sprint: | STM 2022-02-21 | ||||||||||||
| Participants: | |||||||||||||
| Story Points: | 2 | ||||||||||||
| Description |
|
The parallel test suite works by taking tests from the core suite and running them in parallel with each other. For behavior guarded by a feature flag named featureFlagToaster, jstests exercising this new feature can add the tag featureFlagToaster to ensure that the test is only run on all-feature-flags build variants and is excluded from multiversion testing. However, the parallel tester ignores this and attempts to run such tests even on non-feature-flag build variants. Currently, the most simple workaround for this is to simply add each feature flagged test to the list of exclusions for the parallel tester, until the feature flag is enabled by default. However, for certain features this loss in test coverage may not be desirable. If keeping the parallel test coverage is particularly important, another workaround is to explicitly check whether the feature flag is enabled on the server at the beginning of each test. If the server reports that the feature flag is not enabled, the test can simply return early before actually performing any testing. The downside to this is that it is more tedious and partially defeats the purpose of having the featureFlagToaster jstest tag to begin with. |
| Comments |
| Comment by Zituo Jin [ 16/Feb/22 ] |
|
Documentation has been added in the following locations:
|
| Comment by Githook User [ 16/Feb/22 ] |
|
Author: {'name': 'Zituo Jin', 'email': 'zituo.jin@mongodb.com', 'username': 'zituo-jin'}Message: |
| Comment by Zituo Jin [ 14/Feb/22 ] |
|
gregory.noma Based on the data we have collected, out of patch runs on 7761 build (variants), there are 1424 parallel tasks, 72 of them have failed, i.e., 18% of builds run parallel test suite, which has 5% failure rate. If each developer spends 5 minutes looking into task failure, 6 developer hours would be saved with this behavior documented. |
| Comment by Brooke Miller [ 18/Jan/22 ] |
|
To start, we're going to check historic data to get a sense of the signal to noise ratio of the parallel test suite in patch builds. If the ratio is low enough, then we'll advocate for removing this test suite. If the ratio is high, then we're going to follow the below documentation update. We'll post the findings here on the ratio to get confirmation from you (gregory.noma) on the direction to pursue. Documentation update: To support the simple workaround mentioned in the second paragraph in this Description, we'd like to add documentation to explain this workaround in the following locations:
|