[SERVER-68699] Write a test that checks that resmoke suite config is valid Created: 09/Aug/22 Updated: 27/Dec/23 Resolved: 11/Oct/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.2.0-rc0 |
| Type: | Task | Priority: | Minor - P4 |
| Reporter: | Dave Rolsky | Assignee: | Mikhail Shchatko |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Assigned Teams: |
Server Development Platform
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Sprint: | DAG 2023-10-16 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Story Points: | 2 | ||||||||||||||||||||
| Description |
| Comments |
| Comment by Githook User [ 11/Oct/23 ] |
|
Author: {'name': 'Mikhail Shchatko', 'email': 'mikhail.shchatko@mongodb.com', 'username': 'MikhailShchatko'}Message: |
| Comment by Mikhail Shchatko [ 06/Oct/23 ] |
|
I created SERVER-81938 to address tags validation since I see problems in implementing it right now. |
| Comment by Trevor Guidry [ 04/Oct/23 ] |
|
I think you are right, currently resmoke does not have all the context it needs to know what "all tests" is. I think we could add a new config variable similar to our current "config_dir" variable. This new variable would be a list of directories where jstest tests are defined, for us it would default to "jstests" and "src/mongo/db/modules/enterprise/jstests". This new variable would allow us to do two things, recursively search these directories to determine what "all tests" are and only validate suites that use these directories in their selectors. Some suites use non-standard locations and expect certain input. These inputs may or may not be checked in to the git repo and should not be part of this validation. Example: new variable,something like jstest_dirs = ["jstests", "src/mongo/db/modules/enterprise/jstests"] You can scan all tests in these dirs to determine all in use tags You run into suites like jsCore and validate it because the selector paths start with the above dirs You run into suites like generational_fuzzer and do not validate it because it uses non-standard paths Some caveats I know of
Resmoke has a lot of edge cases so I might be missing some stuff, hopefully this was clear and helps but feel free to DM me if you have any questions. |
| Comment by Mikhail Shchatko [ 03/Oct/23 ] |
|
alex.neben@mongodb.com trevor.guidry@mongodb.com Given the conversation above and |
| Comment by Robert Guo (Inactive) [ 11/Aug/22 ] |
|
Got it. Validating roots makes sense. If you'd like, feel free to add a PR with an additional assert here. Regarding maintaining the tags, there would be pre-commit enforcement to ensure tags in files are kept up-to-date. It should be doable to backport to older commits. |
| Comment by Dave Rolsky [ 10/Aug/22 ] |
|
robert.guo@mongodb.com I'd also note that the other part of this ticket is to check that roots is valid. I think that would still be useful. |
| Comment by Dave Rolsky [ 10/Aug/22 ] |
|
robert.guo@mongodb.com, this sounds like a good direction to go. My only question would be how other teams would handle cases where we need to exclude something. How can we get a tag added to a test if it doesn't have what we need? This is further complicated by the fact that right now we are locked into a single older commit of the Server code which we check out to run the resmoke tests with. I'm not sure how hard it is for us to change that commit, though. |
| Comment by Robert Guo (Inactive) [ 09/Aug/22 ] |
|
Hey dave.rolsky@mongodb.com thanks for raising this issue. The interface between mongosync and the server test driver has been brittle and a frequent source of bugs. There was a discussion earlier today about making JS tests more knowable (as part of our dev prod 6 week check in) and we plan to try standardizing tags as a way to make testing more knowable. The work is in The adoption plan after that is to have suites only exclude tests using tags. Do you think this would be a reasonable solution to your pain point? The other side of the story is that the proposed change here is likely fairly involved as resmoke has to support many types of tests and environments and many such environments (e.g. ones with automated test selection), will have some tags being empty. |