[SERVER-28940] Make resmoke fixture setup/teardown their own testcases. Created: 24/Apr/17 Updated: 30/Oct/23 Resolved: 04/Oct/18 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Testing Infrastructure |
| Affects Version/s: | 3.5.6 |
| Fix Version/s: | 4.1.4 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Robert Guo (Inactive) | Assignee: | Yves Duhem |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | tig-qwin-eligible, tig-resmoke | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Sprint: | TIG 2017-07-10, TIG 2018-09-24, DAG 2018-10-08 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Linked BF Score: | 0 | ||||||||||||||||
| Story Points: | 5 | ||||||||||||||||
| Description |
|
The fixture setup and teardown logs only go to logkeeper, but don't show up on the Evergreen sidebar. So the assertions can't be extracted without figuring out the logkeeper URL from the task log. Since everything else that logs to logkeeper (i.e. test and hooks) is a testcase that shows up on the sidebar, the fixture events should get their own spots as well. |
| Comments |
| Comment by Githook User [ 04/Oct/18 ] |
|
Author: {'name': 'Yves Duhem', 'email': 'yves.duhem@mongodb.com', 'username': 'syev'}Message: |
| Comment by Max Hirschhorn [ 19/Sep/18 ] |
|
I've looked back at EVG-1135 and some of the Slack conversations I've had where |
| Comment by Robert Guo (Inactive) [ 18/Sep/18 ] |
|
I had a brief offline discussion with Yves last week. My preference is to add links to the job log to the task logs and make the necessary changes to the build baron tool to parse the job logs. As we add more background hooks, there will be an increasing number of failures that show up outside of the test log of the failing test. Fixture crash stacktraces also sometimes show up earlier than when the crashes are picked up by resmoke's is_running() check. These problems can probably be alleviated if logkeeper adds support for parallel tests, but having the BB tool handle job logs and larger logs in general seems like something we'll still want in the long run. |
| Comment by Yves Duhem [ 18/Sep/18 ] |
|
max.hirschhorn, I agree that having a test for the fixture startup is a straightforward way to get these failures analyzed by the Build Baron tool. It would be more work but it is also possible to modify the Build Baron tool to extract the job URLs from the task log and perform the analysis. |
| Comment by Max Hirschhorn [ 18/Sep/18 ] |
yves.duhem, my issue with that approach is there still wouldn't be a way for the Build Baron tool to analyze the logs associated with a startup failure. |
| Comment by Yves Duhem [ 18/Sep/18 ] |
|
The end goal being to make the job logs more easily accessible when an error occurs during a fixture setup or teardown, robert.guo, max.hirschhorn, what do you think about adding a log line with the URL to the logs when that occurs? It will not be as visible as having a link in the test results but having a test case for the fixture setup and teardown does not really fit in the way the resmoke code is organized and as we are doing more and more analysis on our Evergreen test results I'd prefer to avoid using Evergreen tests for steps that are not strictly speaking tests. |
| Comment by Robert Guo (Inactive) [ 17/Jul/17 ] |
|
Based on discussion with yves.duhem in the code review, the logs from the fixtures won't show up in the fixture startup/teardown testcase, so this ticket is being put on hold for now. |
| Comment by Robert Guo (Inactive) [ 24/Apr/17 ] |
|
max.hirschhorn I was thinking of the teardown phase at the end of a suite, which happens after the last testcase/hook. |
| Comment by Max Hirschhorn [ 24/Apr/17 ] |
|
robert.guo, given that |