[SERVER-70048] Improve developer experience with golden tests Created: 28/Sep/22 Updated: 29/Oct/23 Resolved: 24/Nov/22 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 6.3.0-rc0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Militsa Sotirova | Assignee: | Milena Ivanova |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | greenerbuild | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Backwards Compatibility: | Fully Compatible |
| Participants: |
| Description |
|
When a golden test fails because the actual output didn't match the expected output (as opposed to because of a crash during the test), the output isn't very clear/legible:
It would greatly improve readability if, when this happens, the output dump that follows the error message was either omitted and replaced with a note to run
for more details or omitted and replaced with the output of
directly.
An idea for an approach would be to change the error message that is logged here and/or here to simplify the messages. Also, in the case where a test fails because the actual output didn't match the expected output, the backtrace that is logged doesn't provide any additional information. One way to simplify this would be to create a custom AssertionFailureException class for golden tests that would replace the exception thrown here. This class would not print the stack trace in its constructor like TestAssertionFailure does. |
| Comments |
| Comment by Githook User [ 24/Nov/22 ] |
|
Author: {'name': 'Milena Ivanova', 'email': 'milena.ivanova@mongodb.com', 'username': 'mivanova3'}Message: |
| Comment by David Percy [ 28/Sep/22 ] |
|
Here's where the golden jstests print the diff automatically: https://github.com/10gen/mongo/blob/6da36dce7d1a4fa6c77de6cf8a877d8a58b4f639/src/mongo/shell/shell_utils.cpp#L589-L591 We could move that so both kinds of golden tests use it. |