[SERVER-62557] Parallel tester ignores feature flagging Created: 12/Jan/22  Updated: 09/Jan/23  Resolved: 16/Feb/22

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Gregory Noma Assignee: Zituo Jin
Resolution: Won't Fix Votes: 0
Labels: faster-patches
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-39362 Make parallel/basic.js test respect r... Closed
is related to SERVER-69929 Skip Strict API testing if required f... Closed
Operating System: ALL
Sprint: STM 2022-02-21
Participants:
Story Points: 2

 Description   

The parallel test suite works by taking tests from the core suite and running them in parallel with each other. For behavior guarded by a feature flag named featureFlagToaster, jstests exercising this new feature can add the tag featureFlagToaster to ensure that the test is only run on all-feature-flags build variants and is excluded from multiversion testing. However, the parallel tester ignores this and attempts to run such tests even on non-feature-flag build variants.

Currently, the most simple workaround for this is to simply add each feature flagged test to the list of exclusions for the parallel tester, until the feature flag is enabled by default. However, for certain features this loss in test coverage may not be desirable.

If keeping the parallel test coverage is particularly important, another workaround is to explicitly check whether the feature flag is enabled on the server at the beginning of each test. If the server reports that the feature flag is not enabled, the test can simply return early before actually performing any testing. The downside to this is that it is more tedious and partially defeats the purpose of having the featureFlagToaster jstest tag to begin with.



 Comments   
Comment by Zituo Jin [ 16/Feb/22 ]

Documentation has been added in the following locations:

  • Wiki page for Feature Flagging Lifecycle
  • basic.js and basicPlus.js
  • parallel suite yaml file in resmoke config
Comment by Githook User [ 16/Feb/22 ]

Author:

{'name': 'Zituo Jin', 'email': 'zituo.jin@mongodb.com', 'username': 'zituo-jin'}

Message: SERVER-62557 Add documentation for parallel suite
Branch: master
https://github.com/mongodb/mongo/commit/33199fcbda79a5ec0ec686896583deb460c56028

Comment by Zituo Jin [ 14/Feb/22 ]

gregory.noma Based on the data we have collected, out of patch runs on 7761 build (variants), there are 1424 parallel tasks, 72 of them have failed, i.e., 18% of builds run parallel test suite, which has 5% failure rate. If each developer spends 5 minutes looking into task failure, 6 developer hours would be saved with this behavior documented.

Comment by Brooke Miller [ 18/Jan/22 ]

To start, we're going to check historic data to get a sense of the signal to noise ratio of the parallel test suite in patch builds. If the ratio is low enough, then we'll advocate for removing this test suite. If the ratio is high, then we're going to follow the below documentation update.  We'll post the findings here on the ratio to get confirmation from you (gregory.noma) on the direction to pursue.


Documentation update: To support the simple workaround mentioned in the second paragraph in this Description, we'd like to add documentation to explain this workaround in the following locations:

  • Confluence Wiki page for Feature Flagging Lifecycle 
  • At the end of basic.js, add a try catch and print out that you have to go to paralleltester.js if that test fails 
  • In the resmoke suite yaml file for the parallel suite 
Generated at Thu Feb 08 05:55:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.