[SERVER-63104] Make it easy to run jsCore locally again in a world with tests tagged by feature flag Created: 28/Jan/22 Updated: 29/Oct/23 Resolved: 16/Nov/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | None |
| Fix Version/s: | 6.1.1, 5.0.15, 6.0.4, 6.2.0-rc2, 6.3.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | David Storch | Assignee: | Trevor Guidry |
| Resolution: | Fixed | Votes: | 5 |
| Labels: | vw | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||
| Backport Requested: |
v6.1, v6.0, v5.0
|
||||||||||||
| Participants: | |||||||||||||
| Description |
|
I recently tried to run jsCore locally with my usual invocation:
However, several tests failed:
The reason is that these tests are all tagged with featureFlagCommandsAcceptCollectionUUID. This means that the tests should only be permitted to run when featureFlagCommandsAcceptCollectionUUID is toggled on. However, the local resmoke.py invocation mentioned above will run the server in its default configuration without the feature flags for any in-progress features enabled. It also has no mechanism to exclude tests which are tagged based on feature flags. Ideally, there would be an easy way to run the core test suite locally without a complex command line invocation or any extra steps. At least in my workflow, I do this regularly to confirm that I haven't broken anything fundamental before I move onto running a larger battery of tests. I don't particularly care if this mechanism simulates the release builders or the "all feature flags" builders. In other words, it could either enable all the feature flags (except for the "fully disabled" ones) and run the entire jsCore suite. Alternatively, it could make the method of excluding feature flag tagged tests more automated, so that I didn't need to anything special to exclude tests that won't pass in the server's default configuration. |
| Comments |
| Comment by Githook User [ 16/Nov/22 ] | ||
|
Author: {'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}Message: (cherry picked from commit 22f38cf147eeedca45943ababe818213184eb754) | ||
| Comment by Githook User [ 16/Nov/22 ] | ||
|
Author: {'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}Message: (cherry picked from commit 22f38cf147eeedca45943ababe818213184eb754) | ||
| Comment by Githook User [ 16/Nov/22 ] | ||
|
Author: {'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}Message: (cherry picked from commit 22f38cf147eeedca45943ababe818213184eb754) | ||
| Comment by Githook User [ 16/Nov/22 ] | ||
|
Author: {'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}Message: (cherry picked from commit 22f38cf147eeedca45943ababe818213184eb754) | ||
| Comment by Githook User [ 16/Nov/22 ] | ||
|
Author: {'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}Message: | ||
| Comment by Alex Neben [ 25/Oct/22 ] | ||
|
Hey watchers of this ticket, Note that I am voting against adding a new flag `--genAllFeatureFlags` since that is just more typing for what I suspect will basically be the most common use case. | ||
| Comment by Alex Neben [ 26/Sep/22 ] | ||
|
Seems super reasonable, will prioritize getting that fix in, no promise on date. | ||
| Comment by David Storch [ 23/Sep/22 ] | ||
|
Ah. Thanks Spencer and Steve. Yeah, I agree that it would be nice for this to work "out of the box" so that you don't have to keep downloading a new copy of "all_feature_flags.txt" as the set of feature flags changes. | ||
| Comment by Steve Tarzia [ 23/Sep/22 ] | ||
|
Yea, I'd hope this ticket would allow `–runAllFeatureFlagTests` to work without needing to manually track down and an `all_feature_flags.txt` file that matches the current binary. | ||
| Comment by Spencer Jackson [ 23/Sep/22 ] | ||
|
IIRC, I think you need a copy of all_feature_flags.txt, and you're supposed to extract that file from a recent evergreen run out of archive_dist_test's Artifacts tar file. | ||
| Comment by David Storch [ 23/Sep/22 ] | ||
|
Thanks alexander.neben@mongodb.com! I tried running the following:
But this didn't do what I wanted it to. All of the mongod nodes failed to start up with this error:
I might just be doing something wrong, but is there some kind of artifact that the runAllFeatureFlagTests option needs in order to know what feature flags to enable? What would the steps be to use runAllFeatureFlagTests locally? | ||
| Comment by Alex Neben [ 21/Sep/22 ] | ||
|
Hey I am starting to look into this ticket. I see there is an option `--runAllFeatureFlagTests`. Would running with that solve the issue here? | ||
| Comment by Iryna Zhuravlova [ 09/May/22 ] | ||
|
The ask here is to integrate the feature flag mechanism with SCons so people can use feature flag locally. A longer term decision for which feature flags to run in local JStests can also be brought for into the build system | ||
| Comment by Brooke Miller [ 01/Feb/22 ] | ||
|
We're going to change resmoke's command line option so that it runs with all feature flags by default. |