[langchain-mongodb-deepagents-vfs] Add auto-embedding

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Unknown
    • None
    • Affects Version/s: None
    • Component/s: ABX
    • None
    • None
    • Python Drivers
    • None
    • None
    • None
    • None
    • None
    • None

      Goal

      Support MongoDB Atlas auto-embedding in this package, so Atlas generates and manages embedding vectors server-side and the client no longer calls an embedding API during sync.

      Why this is mostly wiring, not new work

      The monorepo already implements this pattern twice, and the helpers are in pymongo-search-utils. This ticket adapts an established pattern rather than inventing one.

      Existing piece Location
      AutoEmbeddings marker class (model name only; embed_documents/embed_query raise) langchain_mongodb/embeddings.py:6
      Index creation with auto_embedding_model langchain_mongodb/vectorstores.py:1005, langgraph/store/mongodb/base.py:333
      autoembedding_vector_search_stage(query, search_field, index_name, model, ...) pymongo_search_utils/pipeline.py:94
      Param validation: if auto_embedding_model is set, dimensions must be -1 and similarity must be None pymongo_search_utils/index.py:21-26
      Integration tests to model on libs/langchain-mongodb/tests/integration_tests/test_autoembedded_vectorstore.py, libs/langgraph-store-mongodb/tests/integration_tests/test_autoembedded_index.py

      Changes required in this package

      1. Vector index definitionIndexManager._ensure_vector_search_index() currently passes dimensions=self._dims and similarity="cosine". Under auto-embedding it must pass dimensions=-1, similarity=None, auto_embedding_model=<model>, and set path to the text field (content) rather than embedding. Passing the current values alongside auto_embedding_model raises ValueError.
      2. Query pathSearchRouter._grep_hybrid() calls vector_search_stage(query_vector, "embedding", ...). Swap to autoembedding_vector_search_stage(pattern, "content", index_name, model, ...) inside the $rankFusion vector pipeline. The rest of the hybrid pipeline is unchanged.
      3. Sync path — chunk documents are written without an embedding field; Embedder.embed_batch() is not called during sync. The "embedding failed, falling back to full-text only" branch in grep becomes unreachable in auto-embedding mode and should be gated, not left as dead code.
      4. Chunker is still required — auto-embedding embeds a field value subject to per-document token limits, and line_start is what makes GrepMatch.line work. No change here, but state it so it isn't "simplified" away.
      5. Migration — existing collections carry an embedding field and a dimension-based index. Switching modes requires recreating the vector index, not updating it. IndexManager currently has migration logic only for the full-text index.
      6. Filters — the vector index declares filters=["source_path", "filename"]. Confirm filter fields are still supported in an auto-embedding index definition.
      7. Non-Atlas fallback — unchanged; the regex path never touched embeddings.

      What this buys

      • No embedding API credentials, cost, or latency in the sync path.
      • Removes a whole failure class from the watcher — embedding-API failure mid-ingest currently leaves chunks unindexed.
      • The [bedrock] / [openai] extras become optional for Atlas users.

      Acceptance criteria

      • An integration test creates an auto-embedded index, syncs an object, and gets a semantic grep hit without any embedding provider configured.
      • $rankFusion hybrid grep works in auto-embedding mode; scoring/ordering verified against a known corpus.
      • Non-Atlas regex fallback is unaffected.
      • Documents written in auto-embedding mode carry no embedding field.
      • README documents the mode, its Atlas requirement, and the supported-model constraint.

      Decisions needed

      1. Default or opt-in? Options: (a) opt-in via constructor arg, bedrock stays default — safe, no mig; (b) default for new deployments, detect and warn on collections with an existing dimension-based index.There are no existing users yet, since 0.1.0 is unreleased — which argues for making the right choice now rather than a compatible one.
      2. Keep client-side providers at all? Auto-embedding requires Atlas. The non-Atlas fallback already degich needs no embeddings. So client-side providers may only be needed for users on Atlas who want a non-Voyagemodel. Worth deciding whether that is a use case this package supports.
      3. Which model, and how is it configured? Must be one of the Atlas-supported auto-embedding models. Decer it is exposed as a constructor arg, an env var, or both, consistent with the existing {{EMBEDDING_MODEL}}convention.
      4. Interaction with the voyageai provider ticket. Auto-embedding uses Voyage server-side; the siblient-side. Decide whether both ship, and in which order.

      Notes

      • Atlas-only feature — confirm minimum Atlas version/tier and region availability before committing to a tar

            Assignee:
            Unassigned
            Reporter:
            Casey Clements
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: