The current state of project syntax/semantic validation makes little to no sense. I'll try to list the problems here:
1. The patch intent creation job, repotracker, and commit queue only run the project syntax checkers. This means that checks like ensuring the logger is valid do not run, even though having an invalid logger should be something that prevents a patch intent from being created. It also means you can do nonsense things like make tasks that have 0 commands in them and it will still be created (and will also not give you any warning anywhere that your config has problems) since that's listed under the semantic checkers unless you run evergreen validate.
2. There was a slack thread stating that the semantic problems are warnings. We have no way of displaying these warnings to users unless they run evergreen validate.
3. A lot of the validators listed under the syntax checkers are actually semantic checkers, which is highly confusing. This stems from the patch intent problem above where "syntax checking" actually means "error checking". For example, one of the checks that checks there are at most 50 host.create calls should actually be a semantic checker.
I propose the following changes:
- The syntax checker function and semantic checker function should probably be renamed (i.e. change the names to "CheckProjectErrors" and "CheckProjectWarnings") to reflect what they actually do. We should check semantic problems that are legitimate errors but not classify them as "syntax checkers". That way, we don't have this misleading divide between syntax checking and semantic checking.
- Reorganize the check functions according to what's an error and what's a warning.
- Actually check syntax and semantic errors in the patch intent. This can be solved by the renaming and reorganization I described above.
- Display warnings found by the warning checker in the UI somehow so people know that there are problems in their config.