[COMPASS-7292] Run accuracy tests in Compass nightly vs cloud dev Created: 03/Oct/23 Updated: 01/Nov/23 Resolved: 29/Oct/23 |
|
| Status: | Closed |
| Project: | Compass |
| Component/s: | GAI |
| Affects Version/s: | None |
| Fix Version/s: | No version |
| Type: | Task | Priority: | Major - P3 |
| Reporter: | Rhys Howell | Assignee: | Rhys Howell |
| Resolution: | Done | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Story Points: | 3 |
| Documentation Changes: | Not Needed |
| Sprint: | Iteration Minmi, Iteration Nodosaurus |
| Description |
|
We'd like to know when prompt changes or other regressions impact the accuracy of the generative ai results. To do this we'll run the accuracy tests that were recently added to Compass (scripts/ai-accuracy-tests.js) on a nightly basis. They should fail under a certain threshold. |
| Comments |
| Comment by Githook User [ 01/Nov/23 ] |
|
Author: {'name': 'Rhys', 'email': 'Anemy@users.noreply.github.com', 'username': 'Anemy'}Message: chore(compass-generative-ai): add evergreen config for nightly generative-ai accuracy tests |
| Comment by Githook User [ 30/Oct/23 ] |
|
Author: {'name': 'Rhys', 'email': 'Anemy@users.noreply.github.com', 'username': 'Anemy'}Message: chore(compass-generative-ai): add evergreen config for nightly generative-ai accuracy tests |
| Comment by Githook User [ 24/Oct/23 ] |
|
Author: {'name': 'Rhys', 'email': 'Anemy@users.noreply.github.com', 'username': 'Anemy'}Message: chore(compass-generative-ai): add evergreen config for nightly generative-ai accuracy tests |
| Comment by Githook User [ 16/Oct/23 ] |
|
Author: {'name': 'Rhys Howell', 'email': 'rhys.howell@mongodb.com', 'username': 'Anemy'}Message: Merge branch 'main' into |