[SERVER-72606] Filter tests which are incompatible with feature flags provided on the command line Created: 06/Jan/23  Updated: 05/Feb/24

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Matt Broadstone Assignee: Zack Winter
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-73479 resmoke should error locally when att... Closed
Assigned Teams:
Correctness
Participants:

 Description   

Earlier I was trying to debug a failure in the tenant_migration_donor_retry.js test which is marked as incompatible_with_shard_merge. I didn't realize that I was accidentally providing --additionalFeatureFlags featureFlagShardMerge in my local resmoke invocation (a habit because I had been running a lot of tests which require that feature flag recently), so when I ran the test locally I was getting a very different error than in my patch build. This feels like a class of error we should be able to catch programmatically.



 Comments   
Comment by Tausif Rahman (Inactive) [ 07/Apr/23 ]

From yuhong.zhang@mongodb.com via slack: 
For our jstests, can we do something like “exclude this test from these passthroughs only if the feature flag is not enabled”? We do have separate tests running in different setups, but it will be nice to have more coverage with the passthroughs
 

Comment by Tausif Rahman (Inactive) [ 04/Mar/23 ]

We have SERVER-73479 open right now to disallow running "excluded" tests when a test is passed into the command line so that it would require an override.

I don't have all the details of this issue, but if you want to exclude incompatible_with_shard_merge, you would need to pass that in as --excludeWithAnyTags=incompatible_with_shard_merge – which is probably what the evergreen task you were trying to reproduce was using. For resmoke to know that if featureFlagShardMerge is on, we should not run incompatible_with_shard_merge is going to be challenging (let me know if I am missing something here).

For your case, I think we need to improve the local repro experience so that you have all of the same flags locally as the evergreen task runs with in CI. I believe that would have helped you in this situation. I've opened SERVER-74634 for our team to discuss this.

Generated at Thu Feb 08 06:22:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.