Atlas Search evals for compass assistant

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • Developer Tools

      in EAI-1541, we added an eval suite to the compass assistant for tool calls. currently, these eval cases and the surrounding infra only touch core mongodb functionalty, not atlas-only features. 

      however, there's been significant interest in adding atlas search-related evals. we should add a set of >10 atlas search eval cases. collaborate with the search team to get these made. i've previously spoken with ayaan.hussain@mongodb.com about this, with the eval cases to be placed in this table https://docs.google.com/document/d/1w_XaK0gMy1czI9LaHRuWXlBjsEocI4NSBr8xaPvphJU/edit?tab=t.2704vsi5ord4#heading=h.rgq8a5usqkd9 

      note that in order to support the search eval cases, we'll need add a bit of new supporting code:

      1. extend our in-memory mongodb implementation to support atlas search index creation in packages/compass-assistant/test/fixtures/databases/seed-server.ts
      2. add a search index type to packages/compass-assistant/test/types/seed-data.ts
      3. add search indexes to the sample database data defined in packages/compass-assistant/test/fixtures/databases/ 
        1. we can use search indexes already on the sample databases

            Assignee:
            Unassigned
            Reporter:
            Ben Perlmutter
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: