[SERVER-79456] Explore how to easily provide instructions to re-run a failed test Created: 24/Jul/23  Updated: 25/Sep/23  Resolved: 25/Sep/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: Kevin Cherkauer Assignee: Steve Gross
Resolution: Won't Fix Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File nonexistent_suite_names.png    
Assigned Teams:
Server Development Platform
Participants:

 Description   

See this Slack discussion for more detail. Currently the task name is shown at the top level of patch build UI, but I think showing the suite name would be more useful, as suite name is relevant for reproducing the failure via resmoke.py whereas task name is an internally generated name (that often, but not always, matches the suite name).

Maybe there are still reasons not to do this, like it would create too many aliases on the UI display.

 

Original description:

The Evergreen patch build UI shows at least the following nonexistent suite names:

  • noPassthrough
  • noPassthroughWithMongod

The correct names of these two suites are:

  • no_passthrough
  • no_passthrough_with_mongod

Showing nonexistent names makes it hard for developers to reproduce these failures, as they need the correct suite name for the resmoke.py --suite flag or they get a failure like:

[resmoke] 22:50:16.883Z verbatim resmoke.py invocation: build/install/bin/resmoke.py run --installDir build/install/bin --dbpathPrefix /home/ubuntu/data --suite=noPassthroughWithMongod jstests/noPassthroughWithMongod/now_variable.js
...
[resmoke] 22:50:17.041Z Failed to parse YAML suite definition: Unknown suite 'noPassthroughWithMongod'
[resmoke] 22:50:17.041Z Suites available to execute:
aggregation
aggregation_auth
aggregation_column_store_index_passthrough
...
no_passthrough
no_passthrough_with_mongod
...
vector_search
watchdog
write_concern_majority_passthrough
[resmoke] 22:50:17.041Z Exiting with code: 1

There are 369 suites in the output list, making it non-trivial for a developer to find the correct suite.



 Comments   
Comment by Kevin Cherkauer [ 29/Aug/23 ]

trevor.guidry@mongodb.com Thank you for that clarification. I did not know these two flags were really the same. Thus it the needed suite info is in the tab showing the resmoke command, so that problem is solved.

Given that we were both confused by it accepting the shortest unique abbreviation of a flag, it might be worth considering changing the behavior to only accept exact full flag names.

Comment by Kevin Cherkauer [ 22/Aug/23 ]

steve.gross@mongodb.com Thank you for bringing up the larger picture. The use case I am actually pursuing is:

  • Find and locally (i.e. on a VWS) run the individual tests that failed in the patch build.

Normally there is no need to rerun the entire suite, just the specific test(s) that failed, one by one, in order to troubleshoot the failures and test the fixes.

(I can't figure out how to quote a dash-dash-literal here in Jira so it doesn't become strike through text except in a full "noformat" box. References below to "suite flag" are referring to the resmoke.py flag

--suite

)

However, in order to correctly run an individual test that failed, the developer needs the correct suite name to pass to resmoke.py's suite flag. This is because the suite flag triggers the specific setup that matches the particular run that failed in patch build. A given test may be run in many different suites (e.g. standalone, replica sets, sharded, etc. – a lot of tests are run in about 20 different suites out of a universe of hundreds of possible suites), so it is quite common for a test to pass in 19 suites but fail in the 20th. To reproduce and diagnose that failure, the developer needs to provide the same suite flag value that was used in the specific run that failed (e.g. maybe the test passes on standalone and replica set runs but fails in a particular sharding suite).

Most of the time I have found that the name shown as "Name" on the top-level patch build UI screen is the same as the suite flag value being used in that task run. The problem we are trying to solve here is that in the cases where that is not true, how does the developer find the correct suite flag value to use to reproduce the failure on their VWS?

For example, if the test jstests/aggregation/sources/addFields/empty_objs.js failed, the developer needs to issue a command like

build/install/bin/resmoke.py run --suite=SOMETHING jstests/aggregation/sources/addFields/empty_objs.js

where they need to know what the string "SOMETHING" is. Currently in the cases where the patch build's "Name" display name is not the suite name, I do not know how to find this information. That is the specific problem that led me to create this ticket.

Thanks for the whole team's help on this!

Comment by Steve Gross [ 21/Aug/23 ]

Note: I set this to "backlog" since we're at the brainstorming stage.

Comment by Annie Black [ 17/Aug/23 ]

I'm hesitant to move this over to EVG until we really have fleshed this out. I'm seeing this as being heavy lift enough for both of our teams that it's worth exploring other options.

Comment by Annie Black [ 17/Aug/23 ]

This is definitely not simple. The descriptions that you're describing are very server specific; it sounds like the ask is to give you a way to define your own tooltips per task, and additionally that kind of tooltip is really long. Is there an existing documentation for SERVER about Evergreen tasks and how to approach testing? If so, can this be added to that? If not, should there be? 

Comment by Steve Gross [ 14/Aug/23 ]

kevin.cherkauer@mongodb.com Sure, those are very interesting ideas. I've re-assigned this ticket to Evergreen's backlog, since we're really now talking about improvements to the Evergreen experience. Evergreen folks, WDYT?

Comment by Kevin Cherkauer [ 14/Aug/23 ]

steve.gross@mongodb.com The specific request here originated in a misunderstanding on my part of what the "Name" is supposed to be showing in the patch build UI. Since Name nearly always seemed to be identical to the suite name, I thought it actually was the suite name and that therefor the ones showing as different from the true suite name represented a display bug, but it turns out Name is actually an Evergreen task name, not the suite name, and there are suites that get run under many different tasks, making it infeasible to show the suite name as Name.

So we can discard my specific request that originally opened this ticket. Would would be ideal is to have the UI be more transparent, so users (both Server and Evergreen developers) understand at a glance what info is being displayed and how to find the info they need. (E.g. Evergreen task name is not something needed by a developer to run a test on their VWS, but suite name is, and since they are related to each other and often are the same string, it can be confusing.)

Perhaps something like hover help over column headers with explanations as to what is shown and info about how to find related things. Server developers probably don't know what all the Evergreen entities are (like Evergreen task name) so it would be useful, IMO, for hover help to include FAQ style info, like

This is the Evergreen task name. It is often, but not always, the same as the test suite name. To find the suite name, drill down into the bottom-level task (which may be more than one level of drill-down), select the Files tab, and open the "Resmoke.py Invocation for Local Usage" link.

although the above does not always give the suite name either. E.g. this example (from drilling down into the first failure in this patch build shows command

 buildscripts/resmoke.py run
--suites=change_streams_pre_images_replica_sets_kill_secondary_jscore_passthrough
...

which has a -suites flag, not the -suite flag, and these are not the same thing. (Perhaps this gives also an idea of how it may currently be difficult for a Server developer to find the --suite name they need to rerun the test. I have found that commands like the above often do not actually work on a VWS, which is another point of friction in the process.)

From a high-level, it would be great if it were trivially easy to find the --suite name of a failed test run so a Server developer can run with the correct suite on their VWS.

Thank you kindly for your consideration.

 

Comment by Steve Gross [ 08/Aug/23 ]

kevin.cherkauer@mongodb.com : IIUC, you're requesting that a task name be identical to the suite that it runs? If so, I don't think that's feasible: a suite may be run in multiple tasks. As a result, we need unique and different task names to distinguish between the different runs. Or have I misunderstood your request?

Comment by Kevin Cherkauer [ 27/Jul/23 ]

mohamed.khelif@mongodb.com The noPassthrough* names have always been incorrect since I joined last August, as far as I can recall. Not sure what SDP is. Note these are the suite names  (top-level of the display).

Comment by Mohamed Khelif [ 27/Jul/23 ]

Hey kevin.cherkauer@mongodb.com, Evergreen does not dictate what task names are rendered. Could you check with SDP to determine if something changed with the way task names are generated? 

Comment by Kevin Cherkauer [ 24/Jul/23 ]

Screenshot from this patch build
showing a mix of nonexistent (first four) and real (last two) suite names:

Generated at Thu Feb 08 06:41:01 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.