-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
None
-
Developer Tools
in EAI-1541, we added an eval suite to the compass assistant for tool calls. currently, these eval cases and the surrounding infra only touch core mongodb functionalty, not atlas-only features.
however, there's been significant interest in adding atlas search-related evals. we should add a set of >10 atlas search eval cases. collaborate with the search team to get these made. i've previously spoken with ayaan.hussain@mongodb.com about this, with the eval cases to be placed in this table https://docs.google.com/document/d/1w_XaK0gMy1czI9LaHRuWXlBjsEocI4NSBr8xaPvphJU/edit?tab=t.2704vsi5ord4#heading=h.rgq8a5usqkd9
note that in order to support the search eval cases, we'll need add a bit of new supporting code:
- extend our in-memory mongodb implementation to support atlas search index creation in packages/compass-assistant/test/fixtures/databases/seed-server.ts
- add a search index type to packages/compass-assistant/test/types/seed-data.ts
- add search indexes to the sample database data defined in packages/compass-assistant/test/fixtures/databases/
- we can use search indexes already on the sample databases