[SERVER-66926] Introduce end-to-end functional query correctness testing based on golden test framework Created: 01/Jun/22  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Minor - P4
Reporter: David Storch Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-67415 Expose golden-test framework to jstests Closed
Assigned Teams:
Query Optimization
Sprint: QE 2022-06-13, QE 2022-06-27
Participants:

 Description   

Add a new suite called something like query_integration_tests_standalone. This would work much like the pre-existing integration_tests_standalone, but it would be focused on testing the correctness of query results or potentially even testing plans using explain. Critically, the idea is to use golden file based testing – the tests would run some queries or explain operations, dump their results, and then compare the results to the golden files. Having an integration test suite like this is an important part of our testing strategy for the Common Query Framework. It could also strengthen our query correctness testing in general – the current testing is either based on the generational fuzzer which may not generate realistic queries, or based on making specific assertions about the nature of the query results which are weaker tests.



 Comments   
Comment by David Percy [ 06/Jul/22 ]

David Percy, does SERVER-67415 include introducing a new test suite which is supposed to contain only tests which use the golden framework (in JS) to check query correctness?

Yes, I'm planning to add a new suite for these tests. Maybe jstests/query_golden/.

I do believe that we should isolate the golden JS tests into such a suite to avoid the perils that passthrough testing poses to a golden data based approach.

I agree we shouldn't enable any sharding/repl passthroughs yet. But I think we should at least run these new tests against both classic and CQF: running against classic helps us make sure the tests / expected output are correct.

It is important though for each test author to avoid making brittle assertions, like assuming an order for find() without a sort().

Comment by David Storch [ 23/Jun/22 ]

david.percy@mongodb.com is working on related ticket SERVER-67415, so I don't think there is an immediate need for this ticket. The idea of SERVER-67415 is similar, but allows the test author to write JS executed in the mongo shell rather than writing a C++ integration test. I'm going to move this ticket into the triage queue since I'm not entirely sure where it should live.

david.percy@mongodb.com, does SERVER-67415 include introducing a new test suite which is supposed to contain only tests which use the golden framework (in JS) to check query correctness? I do believe that we should isolate the golden JS tests into such a suite to avoid the perils that passthrough testing poses to a golden data based approach.

Generated at Thu Feb 08 06:06:49 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.