[COMPASS-7292] Run accuracy tests in Compass nightly vs cloud dev Created: 03/Oct/23  Updated: 01/Nov/23  Resolved: 29/Oct/23

Status: Closed
Project: Compass
Component/s: GAI
Affects Version/s: None
Fix Version/s: No version

Type: Task Priority: Major - P3
Reporter: Rhys Howell Assignee: Rhys Howell
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Story Points: 3
Documentation Changes: Not Needed
Sprint: Iteration Minmi, Iteration Nodosaurus

 Description   

We'd like to know when prompt changes or other regressions impact the accuracy of the generative ai results. To do this we'll run the accuracy tests that were recently added to Compass (scripts/ai-accuracy-tests.js) on a nightly basis. They should fail under a certain threshold. 
Currently these tests might be synchronous, we might need to parallelize if its really slow.



 Comments   
Comment by Githook User [ 01/Nov/23 ]

Author:

{'name': 'Rhys', 'email': 'Anemy@users.noreply.github.com', 'username': 'Anemy'}

Message: chore(compass-generative-ai): add evergreen config for nightly generative-ai accuracy tests COMPASS-7292 (#4985)
Branch: bulk-update-modal
https://github.com/mongodb-js/compass/commit/146e50dcef71726518d7552adb57591b582cad03

Comment by Githook User [ 30/Oct/23 ]

Author:

{'name': 'Rhys', 'email': 'Anemy@users.noreply.github.com', 'username': 'Anemy'}

Message: chore(compass-generative-ai): add evergreen config for nightly generative-ai accuracy tests COMPASS-7292 (#4985)
Branch: beta-releases
https://github.com/mongodb-js/compass/commit/146e50dcef71726518d7552adb57591b582cad03

Comment by Githook User [ 24/Oct/23 ]

Author:

{'name': 'Rhys', 'email': 'Anemy@users.noreply.github.com', 'username': 'Anemy'}

Message: chore(compass-generative-ai): add evergreen config for nightly generative-ai accuracy tests COMPASS-7292 (#4985)
Branch: main
https://github.com/mongodb-js/compass/commit/146e50dcef71726518d7552adb57591b582cad03

Comment by Githook User [ 16/Oct/23 ]

Author:

{'name': 'Rhys Howell', 'email': 'rhys.howell@mongodb.com', 'username': 'Anemy'}

Message: Merge branch 'main' into COMPASS-7292-run-ai-accuracy-tests
Branch: COMPASS-7292-run-ai-accuracy-tests
https://github.com/mongodb-js/compass/commit/cb6b298cc3acc907d178023fd9eb696088884f09

Generated at Wed Feb 07 22:46:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.