[SERVER-70048] Improve developer experience with golden tests Created: 28/Sep/22  Updated: 29/Oct/23  Resolved: 24/Nov/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.3.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Militsa Sotirova Assignee: Milena Ivanova
Resolution: Fixed Votes: 0
Labels: greenerbuild
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Screen Shot 2022-09-27 at 4.34.31 PM.png    
Backwards Compatibility: Fully Compatible
Participants:

 Description   

When a golden test fails because the actual output didn't match the expected output (as opposed to because of a crash during the test), the output isn't very clear/legible:

It would greatly improve readability if, when this happens, the output dump that follows the error message was either omitted and replaced with a note to run 

golden diff

 for more details or omitted and replaced with the output of 

golden diff

 directly.

 

An idea for an approach would be to change the error message that is logged here and/or here to simplify the messages.

Also, in the case where a test fails because the actual output didn't match the expected output, the backtrace that is logged doesn't provide any additional information. One way to simplify this would be to create a custom AssertionFailureException class for golden tests that would replace the exception thrown here. This class would not print the stack trace in its constructor like TestAssertionFailure does.



 Comments   
Comment by Githook User [ 24/Nov/22 ]

Author:

{'name': 'Milena Ivanova', 'email': 'milena.ivanova@mongodb.com', 'username': 'mivanova3'}

Message: SERVER-70048 Improve developer experience with golden tests
Branch: master
https://github.com/mongodb/mongo/commit/2bc320ca782cd7f7759d8734ee3c29bcc0af9a47

Comment by David Percy [ 28/Sep/22 ]

Here's where the golden jstests print the diff automatically: https://github.com/10gen/mongo/blob/6da36dce7d1a4fa6c77de6cf8a877d8a58b4f639/src/mongo/shell/shell_utils.cpp#L589-L591

We could move that so both kinds of golden tests use it.

Generated at Thu Feb 08 06:15:08 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.