Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-4307

Investigate intermittent failures in LlamaIndex and ChatGPT Retrieval Plugin

    • Type: Icon: Task Task
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 4.7
    • Affects Version/s: None
    • Component/s: AI/ML
    • Labels:
      None
    • Python Drivers
    • Not Needed
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      Context

      We have seen test failures that are suspicious

      1. chatgpt-retrieval-plugin. Results imply that the vector search is returning less that "limit" which is the number of documents requested. I have spoken to Trevor M on #vector-search-engineering who said this should not happen.
        1. I saw this locally once, when I ran in PyCharm upon first running after leaving the test for some time, but additional runs wouldn't reproduce. (Apologies if that is confusing.)
      2. llama-index. We get different answers to a query.
        1. I have seen this also happen just once locally, but not reproduce-ably.
        2. Could it be that the vector search is returning different results? I have to dig into the algorithm, but it does have random components.  

       

      Definition of done

      What must be done to consider the task complete?

      Pitfalls

      What should the implementer watch out for? What are the risks?

            Assignee:
            casey.clements@mongodb.com Casey Clements
            Reporter:
            casey.clements@mongodb.com Casey Clements
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: