-
Type: Improvement
-
Resolution: Won't Fix
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Testing Infrastructure
-
Correctness
-
3
When a task fails in evergreen, a common first step for a developer to triage is to run the failing test locally. Today, this is mostly seamless as task names map to suites 1-1 (with no special configuration required: looking at you --storageEngine=mmapv1 being the default on 4.0 tests)
This isn't entirely defined as what's problematic, but the most annoying one for me is the core/jsCore difference
The other class of issues are tasks that also specify additional arguments such as _compatibility tasks:
resmoke_args: --suites=core --shellReadMode=legacy --shellWriteMode=compatibility --storageEngine=wiredTiger --excludeWithAnyTags=requires_find_command
or the "concurrent" variety of tasks that use --numClientsPerFixture, or tasks that specify something other than wiredTiger, such as the watchdog_* tasks.
Additionally, when a variant adds flags, that's an extra layer of denormalization a developer has to do.
Solving the first issue could be as simple as doing a pass over what's inconsistent and renaming the task or the suite (ideally with some alias thing that wouldn't break existing users that have committed to memorizing the differences?)
Not sure what the options are for the second problem. It's probably best tackled with a combination of changes that can alleviate specific pain points.