-
Type:
Task
-
Resolution: Done
-
Priority:
Unknown
-
None
-
Affects Version/s: None
-
Component/s: ai-ml-testing, AI/ML, CrewAI
-
None
-
None
-
Python Drivers
-
Not Needed
-
-
None
-
None
-
None
-
None
-
None
-
None
Context
The test added in INTPYTHON-580 is flaky. It has failed after merging in two different ways:
In https://spruce.mongodb.com/task/ai_ml_pipeline_testing_test_crewai_tools_rhel_test_crewai_tools_local_3c665f00011e4e657eaf10847c63bdcfbe7422d9_25_07_14_18_08_54/logs?execution=0 we get the error:
[2025/07/14 13:14:21.381] Traceback (most recent call last): [2025/07/14 13:14:21.381] File "/data/mci/45241060f7cd5b50273ddb8d8d569150/src/crewai-tools/crewAI-tools/test_mongodb_vector_search_tool.py", line 78, in <module> [2025/07/14 13:14:21.381] assert "advancements" in text or "improvements" in text, text [2025/07/14 13:14:21.381] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2025/07/14 13:14:21.381] AssertionError: gpt-4 has shown to have remarkable accuracy and capabilities compared to its predecessors....
In https://spruce.mongodb.com/task/ai_ml_pipeline_testing_test_crewai_tools_rhel_test_crewai_tools_remote_68759680cd7ebf000743298b_25_07_14_23_45_04/logs?execution=0 we get the error:
[2025/07/14 18:49:58.248] Traceback (most recent call last): [2025/07/14 18:49:58.248] File "/data/mci/d27c5bb6761b8a88cc694d6c7febeff0/src/crewai-tools/crewAI-tools/test_mongodb_vector_search_tool.py", line 78, in <module> [2025/07/14 18:49:58.248] assert "advancements" in text or "improvements" in text, text [2025/07/14 18:49:58.248] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [2025/07/14 18:49:58.248] AssertionError: i was unable to extract specific insights regarding the accuracy of gpt-4 from the technical document as the tool provided did not allow for successful queries
We could add more expected words, but it wouldn't solve the second problem.
We could try changing the model used, or worst case add a retry.
Definition of done
Address the flaky test.
Pitfalls
None