[SERVER-66926] Introduce end-to-end functional query correctness testing based on golden test framework Created: 01/Jun/22 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Minor - P4 |
| Reporter: | David Storch | Assignee: | Backlog - Query Optimization |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Query Optimization
|
||||||||
| Sprint: | QE 2022-06-13, QE 2022-06-27 | ||||||||
| Participants: | |||||||||
| Description |
|
Add a new suite called something like query_integration_tests_standalone. This would work much like the pre-existing integration_tests_standalone, but it would be focused on testing the correctness of query results or potentially even testing plans using explain. Critically, the idea is to use golden file based testing – the tests would run some queries or explain operations, dump their results, and then compare the results to the golden files. Having an integration test suite like this is an important part of our testing strategy for the Common Query Framework. It could also strengthen our query correctness testing in general – the current testing is either based on the generational fuzzer which may not generate realistic queries, or based on making specific assertions about the nature of the query results which are weaker tests. |
| Comments |
| Comment by David Percy [ 06/Jul/22 ] |
Yes, I'm planning to add a new suite for these tests. Maybe jstests/query_golden/.
I agree we shouldn't enable any sharding/repl passthroughs yet. But I think we should at least run these new tests against both classic and CQF: running against classic helps us make sure the tests / expected output are correct. It is important though for each test author to avoid making brittle assertions, like assuming an order for find() without a sort(). |
| Comment by David Storch [ 23/Jun/22 ] |
|
david.percy@mongodb.com is working on related ticket david.percy@mongodb.com, does |