[SERVER-70718] Resmoke on error should float the most important info to end of log. Created: 19/Oct/22 Updated: 29/Oct/23 Resolved: 21/Feb/23 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 7.0.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Charlie Swanson | Assignee: | Tausif Rahman (Inactive) |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Server Development Platform
|
||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||
| Participants: | |||||||||
| Description |
|
It should be easy to find the error in the resmoke log. Most of the time we should be able to see the problem as the last thing in the evergreen/resmoke log.
For local development we still want this |
| Comments |
| Comment by Githook User [ 18/Feb/23 ] |
|
Author: {'name': 'Tausif Rahman', 'email': 'tausif.rahman@mongodb.com', 'username': 'trahman1318'}Message: |
| Comment by Tausif Rahman (Inactive) [ 02/Feb/23 ] |
|
Sounds great! I looked into this a little and It looks like resmoke continually "reports" data pre/post test. We might be able to store interesting logs to the "report object" upon stopTest or something similar. I think we have a good path forward & will put out a PR soon. |
| Comment by Charlie Swanson [ 02/Feb/23 ] |
|
tausif.rahman@mongodb.com thanks for thinking it through! 1. That's a good point and I think it's OK to only do it if --continueOnFailure is not specified, or to have it stop trying after spewing ~1000 lines of logs or something like that 2. This is worth trying to avoid, but anything other than "the MongoDs all shut down at the end of the test" is better I think 3. Yes this is a reasonable take and one that would give flexibility for it to evolve over time. In fact, I use the bb-log-analyzer tool myself locally: https://github.com/10gen/employees/blob/master/home/charlie.swanson/mongofiles/resmoke.sh One concern and pain point I've had with this is that it takes a while to run after the test fails. I would still like to explore whether we can know in resmoke.py better which logs are relevant before we blow past them. For example, if a test fails, could we capture the previous 100 lines, save them, then shut down the test fixtures, then print those lines again? Or something like that - just spitballing. Maybe we could extract some of the code from the bb-log-analyzers and re-use it inside of resmoke to do it live when the test fails? And maybe that will speed it up? |
| Comment by Tausif Rahman (Inactive) [ 02/Feb/23 ] |
|
I see the convenience factor of this. In resmoke, the last logs tend to be "TestXYZ failed – go look at those logs to investigate this" with no indication of what the issue could be. Then the dev has to go scroll/search for this test failure and filter out the noise. Instead, we'd like those logs right at the end – or at least some idea of what went wrong. Does that sound right? A couple of questions: 1. Will this get noisy if we have --continueOnFailure & there end up being a lot of failures? ie: If we run 100 tests and they all fail, our logs will look like [resmoke test logs][new additional logging] – where [new additional logging] can be huge. Or maybe this is not an issue because devs typically run 1 test at a time or do not run with --continueOnFailure. We could also only float the errors when --continueOnFailure is not passed. 2. What if the [new additional logging] is not sufficient or misleading for investigation? Then did we just add noise, or is it still net positive to have "some" idea of what failed? 3. This is kind of getting into the implementation details but I'm not sure how effectively we will be able to get js stacktraces for processes launched by resmoke. I think the best we will be able to do is some regex on the js logs after a test fails. The results may be similar to what is seen in build baron tools: https://buildbaron.corp.mongodb.com/ui/#/bfg/BFG-1793935 . Is that desirable? |
| Comment by Alex Neben [ 28/Jan/23 ] |
|
I am not totally sure what is involved with this ask or how hard it will be to do. But could you investigate it and see if this is something that is reasonable to do? |