[SERVER-67724] Consolidate all CQF related tests under a resmoke command Created: 01/Jul/22  Updated: 29/Oct/23  Resolved: 16/Aug/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.1.0-rc0

Type: Task Priority: Major - P3
Reporter: Timour Katchaounov Assignee: Charlie Swanson
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to SERVER-68063 Ability to define a sub-set of unit t... Closed
Backwards Compatibility: Fully Compatible
Sprint: QO 2022-07-11, QO 2022-07-25, QO 2022-08-08, QO 2022-08-22
Participants:

 Description   

While working on the CQF project, it is expected that changes to optimizer-related areas will cause multiple changes in many CQF-related tests. Developers should be able to quickly and conveniently run all tests specifically written to test CQF in order to quickly isolate and investigate these changes, update expected results, and rerun the same tests.

This iterative process should be done locally on each developer's machine before starting a more extensive test on Evergreen.

Ideally CQF tests should be run via one or two resmoke commands (one for unit tests, and one for JS tests).

The addition of new tests should either happen automatically by virtue of adding them to certain directories, or there should be clear documentation how to do that, and it should be a requirement for each CQF task to make sure new tests are added to the CQF-specific setup.



 Comments   
Comment by Githook User [ 16/Aug/22 ]

Author:

{'name': 'Charlie Swanson', 'email': 'charlie.swanson@mongodb.com', 'username': 'cswanson310'}

Message: SERVER-67724 Document recommended set of bonsai tests
Branch: master
https://github.com/mongodb/mongo/commit/6e72559c750956215fdf9db5753265a0e74fce8d

Comment by Charlie Swanson [ 26/Jul/22 ]

Looks like it's going to be difficult to do much better on the unit tests until https://jira.mongodb.org/browse/SERVER-68063 is resolved. That ticket isn't scheduled so let's not hold our breath. Without anything on the unit tests front I don't think there's much to commit for this ticket. The command above (amended as Hana suggests) is about as best as I can come up with. I do think it is helpful to maintain this list somewhere for people to update - should we start a wiki page maybe? Or stick a README inside of src/mongo/db/query/optimizer? 

Comment by Hana Pearlman [ 26/Jul/22 ]

As part of SERVER-66265 we added a new suite to run cqf tests with pipeline optimizations disabled: "cqf_disabled_pipeline_opt". So we can update the suites to

--suites=cqf,cqf_parallel,cqf_passthrough,cqf_disabled_pipeline_opt

for now.

Comment by Charlie Swanson [ 14/Jul/22 ]

This works for now:

ninja <FLAGS> install-devcore build/install/bin/algebra_test build/install/bin/db_pipeline_test build/install/bin/optimizer_test build/install/bin/sbe_abt_test && ./build/install/bin/algebra_test && ./build/install/bin/db_pipeline_test --fileNameFilter=pipeline_test.cpp && ./build/install/bin/optimizer_test && ./build/install/bin/sbe_abt_test && resmoke --suites=cqf,cqf_parallel,cqf_passthrough -j4

It's not ideal, and I think it runs both versions of "pipeline_test.cpp" instead of just abt/pipeline_test.cpp, but it'll do as a crutch.

Comment by Timour Katchaounov [ 07/Jul/22 ]

How about unit tests?

Comment by David Percy [ 06/Jul/22 ]

That list sounds good! I forgot we have more than one suite already. I don't have another suite yet; I was thinking we may add one soon, which uses the golden-data framework.

Comment by Charlie Swanson [ 06/Jul/22 ]

david.percy@mongodb.com yes you can! -suites accepts a comma separated list. Did you have specific examples in mind? I'm working on the shortest incantation for the unit tests, but I was just thinking -suites=cqf,cqf_parallel,core_cqf_passthrough for the jstests

Comment by David Percy [ 06/Jul/22 ]

Can we create a single resmoke command that runs several suites?

The reason I ask is, we may want more than one suite of JS tests. For example: one suite for cqf-specific behavior, and one suite that must pass in both classic and cqf.

Generated at Thu Feb 08 06:08:53 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.