[SERVER-28940] Make resmoke fixture setup/teardown their own testcases. Created: 24/Apr/17  Updated: 30/Oct/23  Resolved: 04/Oct/18

Status: Closed
Project: Core Server
Component/s: Testing Infrastructure
Affects Version/s: 3.5.6
Fix Version/s: 4.1.4

Type: Improvement Priority: Major - P3
Reporter: Robert Guo (Inactive) Assignee: Yves Duhem
Resolution: Fixed Votes: 1
Labels: tig-qwin-eligible, tig-resmoke
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-40592 Uncaught exception in resmoke.py job ... Closed
related to SERVER-42356 teardown(finished=True) isn't ever ca... Closed
related to SERVER-42622 resmoke.py doesn't attempt to tear th... Closed
Backwards Compatibility: Fully Compatible
Sprint: TIG 2017-07-10, TIG 2018-09-24, DAG 2018-10-08
Participants:
Linked BF Score: 0
Story Points: 5

 Description   

The fixture setup and teardown logs only go to logkeeper, but don't show up on the Evergreen sidebar. So the assertions can't be extracted without figuring out the logkeeper URL from the task log.

Since everything else that logs to logkeeper (i.e. test and hooks) is a testcase that shows up on the sidebar, the fixture events should get their own spots as well.



 Comments   
Comment by Githook User [ 04/Oct/18 ]

Author:

{'name': 'Yves Duhem', 'email': 'yves.duhem@mongodb.com', 'username': 'syev'}

Message: SERVER-28940 Test cases for fixture setups and teardowns
Branch: master
https://github.com/mongodb/mongo/commit/1e026f75dc41cca4c7293e42b5b49cb9e46d0ea3

Comment by Max Hirschhorn [ 19/Sep/18 ]

I've looked back at EVG-1135 and some of the Slack conversations I've had where SERVER-28940 has explicitly come up. I'm amending my earlier answer and will say that I don't think writing a log message to the task logs (even if it links to the /all logs) is a worthwhile experience for Server engineers, as it isn't much different than how the job logs urls are written to task logs by resmoke. already. The problem this ticket is aiming to resolve with MongoDB startup failures is how Server engineers aren't well-trained in when they should look at the task logs; the ideal experience is to have something for Server engineers to click on in the "Test Results" pane. Addressing that pain point for Server engineers should make the Build Baron tool more useful for "free".

Comment by Robert Guo (Inactive) [ 18/Sep/18 ]

I had a brief offline discussion with Yves last week. My preference is to add links to the job log to the task logs and make the necessary changes to the build baron tool to parse the job logs. As we add more background hooks, there will be an increasing number of failures that show up outside of the test log of the failing test. Fixture crash stacktraces also sometimes show up earlier than when the crashes are picked up by resmoke's is_running() check. These problems can probably be alleviated if logkeeper adds support for parallel tests, but having the BB tool handle job logs and larger logs in general seems like something we'll still want in the long run.

Comment by Yves Duhem [ 18/Sep/18 ]

max.hirschhorn, I agree that having a test for the fixture startup is a straightforward way to get these failures analyzed by the Build Baron tool.

It would be more work but it is also possible to modify the Build Baron tool to extract the job URLs from the task log and perform the analysis.

Comment by Max Hirschhorn [ 18/Sep/18 ]

It will not be as visible as having a link in the test results but having a test case for the fixture setup and teardown does not really fit in the way the resmoke code is organized and as we are doing more and more analysis on our Evergreen test results I'd prefer to avoid using Evergreen tests for steps that are not strictly speaking tests.

yves.duhem, my issue with that approach is there still wouldn't be a way for the Build Baron tool to analyze the logs associated with a startup failure.

Comment by Yves Duhem [ 18/Sep/18 ]

The end goal being to make the job logs more easily accessible when an error occurs during a fixture setup or teardown, robert.guo, max.hirschhorn, what do you think about adding a log line with the URL to the logs when that occurs?

It will not be as visible as having a link in the test results but having a test case for the fixture setup and teardown does not really fit in the way the resmoke code is organized and as we are doing more and more analysis on our Evergreen test results I'd prefer to avoid using Evergreen tests for steps that are not strictly speaking tests.

Comment by Robert Guo (Inactive) [ 17/Jul/17 ]

Based on discussion with yves.duhem in the code review, the logs from the fixtures won't show up in the fixture startup/teardown testcase, so this ticket is being put on hold for now.

Comment by Robert Guo (Inactive) [ 24/Apr/17 ]

max.hirschhorn I was thinking of the teardown phase at the end of a suite, which happens after the last testcase/hook.

Comment by Max Hirschhorn [ 24/Apr/17 ]

robert.guo, given that SERVER-26522 made the CleanEveryN hook into its own test case, is this really only for the initial setup of the fixtures?

Generated at Thu Feb 08 04:19:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.