[SERVER-63104] Make it easy to run jsCore locally again in a world with tests tagged by feature flag Created: 28/Jan/22  Updated: 29/Oct/23  Resolved: 16/Nov/22

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: None
Fix Version/s: 6.1.1, 5.0.15, 6.0.4, 6.2.0-rc2, 6.3.0-rc0

Type: Improvement Priority: Major - P3
Reporter: David Storch Assignee: Trevor Guidry
Resolution: Fixed Votes: 5
Labels: vw
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Related
related to SERVER-62992 Remove need for resmoke.ini Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v6.1, v6.0, v5.0
Participants:

 Description   

I recently tried to run jsCore locally with my usual invocation:

python3 buildscripts/resmoke.py run --suites=core -j 40 --continueOnFailure

However, several tests failed:

jstests/core/collection_uuid_find.js
jstests/core/collection_uuid_write_commands.js
jstests/core/collection_uuid_index_commands.js
jstests/core/collection_uuid_coll_mod.js
jstests/core/collection_uuid_rename_collection.js

The reason is that these tests are all tagged with featureFlagCommandsAcceptCollectionUUID. This means that the tests should only be permitted to run when featureFlagCommandsAcceptCollectionUUID is toggled on. However, the local resmoke.py invocation mentioned above will run the server in its default configuration without the feature flags for any in-progress features enabled. It also has no mechanism to exclude tests which are tagged based on feature flags.

Ideally, there would be an easy way to run the core test suite locally without a complex command line invocation or any extra steps. At least in my workflow, I do this regularly to confirm that I haven't broken anything fundamental before I move onto running a larger battery of tests.

I don't particularly care if this mechanism simulates the release builders or the "all feature flags" builders. In other words, it could either enable all the feature flags (except for the "fully disabled" ones) and run the entire jsCore suite. Alternatively, it could make the method of excluding feature flag tagged tests more automated, so that I didn't need to anything special to exclude tests that won't pass in the server's default configuration.



 Comments   
Comment by Githook User [ 16/Nov/22 ]

Author:

{'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}

Message: SERVER-63104 add resmoke argument for generating all_feature_flags.txt locally

(cherry picked from commit 22f38cf147eeedca45943ababe818213184eb754)
Branch: v6.2
https://github.com/mongodb/mongo/commit/ee271f84ba2a46ce6ead81d4be21e075c4ba69ca

Comment by Githook User [ 16/Nov/22 ]

Author:

{'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}

Message: SERVER-63104 add resmoke argument for generating all_feature_flags.txt locally

(cherry picked from commit 22f38cf147eeedca45943ababe818213184eb754)
Branch: v6.1
https://github.com/mongodb/mongo/commit/5171d72f3a20559f7433d4a6a7283de189cb81a2

Comment by Githook User [ 16/Nov/22 ]

Author:

{'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}

Message: SERVER-63104 add resmoke argument for generating all_feature_flags.txt locally

(cherry picked from commit 22f38cf147eeedca45943ababe818213184eb754)
Branch: v6.0
https://github.com/mongodb/mongo/commit/72063954fe932e99fcd94bb5b30fe18345a14291

Comment by Githook User [ 16/Nov/22 ]

Author:

{'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}

Message: SERVER-63104 add resmoke argument for generating all_feature_flags.txt locally

(cherry picked from commit 22f38cf147eeedca45943ababe818213184eb754)
Branch: v5.0
https://github.com/mongodb/mongo/commit/8edd1125d0311b4136f86d8809201336e39da698

Comment by Githook User [ 16/Nov/22 ]

Author:

{'name': 'Trevor Guidry', 'email': 'trevor.guidry@mongodb.com'}

Message: SERVER-63104 add resmoke argument for generating all_feature_flags.txt locally
Branch: master
https://github.com/mongodb/mongo/commit/22f38cf147eeedca45943ababe818213184eb754

Comment by Alex Neben [ 25/Oct/22 ]

Hey watchers of this ticket,
In this PR I am suggesting an API change. `--runAllFeatureFlagTests` will now automatically compute `all_feature_flags.txt`. However, with this change you would no longer be able to use `--runAllFeatureFlagTests` with a file of user chosen feature flags. Does anyone depend on that functionality? Please reply if you depend on that functionality.

Note that I am voting against adding a new flag `--genAllFeatureFlags` since that is just more typing for what I suspect will basically be the most common use case.

Comment by Alex Neben [ 26/Sep/22 ]

Seems super reasonable, will prioritize getting that fix in, no promise on date.

Comment by David Storch [ 23/Sep/22 ]

Ah. Thanks Spencer and Steve. Yeah, I agree that it would be nice for this to work "out of the box" so that you don't have to keep downloading a new copy of "all_feature_flags.txt" as the set of feature flags changes.

Comment by Steve Tarzia [ 23/Sep/22 ]

Yea, I'd hope this ticket would allow `–runAllFeatureFlagTests` to work without needing to manually track down and an `all_feature_flags.txt` file that matches the current binary.

Comment by Spencer Jackson [ 23/Sep/22 ]

IIRC, I think you need a copy of all_feature_flags.txt, and you're supposed to extract that file from a recent evergreen run out of archive_dist_test's Artifacts tar file.

Comment by David Storch [ 23/Sep/22 ]

Thanks alexander.neben@mongodb.com! I tried running the following:

python3 buildscripts/resmoke.py run --suites=core -j 40 --continueOnFailure --runAllFeatureFlagTests

But this didn't do what I wanted it to. All of the mongod nodes failed to start up with this error:

[j15] {"t":{"$date":"2022-09-23T19:28:43.571Z"},"s":"F",  "c":"CONTROL",  "id":20574,   "ctx":"main","msg":"Error during global initialization","attr":{"error":{"code":2,"codeName":"BadValue","errmsg":"Unknown --setParameter 'featureFlagChangeStreamPreAndPostImages'"}}}

I might just be doing something wrong, but is there some kind of artifact that the runAllFeatureFlagTests option needs in order to know what feature flags to enable? What would the steps be to use runAllFeatureFlagTests locally?

Comment by Alex Neben [ 21/Sep/22 ]

Hey I am starting to look into this ticket. I see there is an option `--runAllFeatureFlagTests`. Would running with that solve the issue here?

Comment by Iryna Zhuravlova [ 09/May/22 ]

The ask here is to integrate the feature flag mechanism with SCons so people can use feature flag locally. A longer term decision for which feature flags to run in local JStests can also be brought for into the build system

Comment by Brooke Miller [ 01/Feb/22 ]

We're going to change resmoke's command line option so that it runs with all feature flags by default. 

Generated at Thu Feb 08 05:56:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.