Add $rankFusion to Mastra

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None

      Two upstream contributions to `mastra-ai/mastra`, both merged and released:

      Use Case

      As a... developer building on MongoDB Atlas with Mastra, including the MongoDB + Mastra agentic AI quickstart.

      I want... `MongoDBVector` to index a collection I already own without Mastra creating, mutating, or dropping it, and to reach MongoDB's full-text (`$search`) and hybrid (`$rankFusion`) retrieval through the store rather than only `$vectorSearch`. And `@mastra/voyageai` to send a multimodal payload the API accepts and to take a `baseUrl` so it can authenticate against MongoDB's hosted Voyage endpoint.

      So that... the quickstart can use MongoDB's own retrieval primitives on existing data without dropping out of Mastra to the raw driver and raw SDK.

      User Experience

      • `createIndex( { collectionName, allowWrites? }

        )` points a Mastra index at an existing collection. Mastra never creates or drops it. Writes throw unless `allowWrites: true`, so read-only is the default.

      • `createSearchIndex` + `textQuery` give BM25. `hybridQuery` fuses vector and text server-side with `$rankFusion` (MongoDB 8.0+).
      • `metadataMode: 'document'` returns the full source document; `ObjectId` `_id`s round-trip as strings.
      • `@mastra/voyageai` multimodal text embedding returned HTTP 400 on every call before this, and had no way to reach a provider-hosted endpoint. Both now work.
      • Backward compatible: all new parameters are optional, and omitting them gives identical prior behavior.

      Dependencies

      • No upstream work is pending. Both PRs are merged and both packages are published.
      • `hybridQuery` needs MongoDB >= 8.0 for `$rankFusion`. Verified on Atlas 8.0.28.
      • #19804 bumps `voyageai` `^0.2.1` to `^0.3.1`. The bump is required: 0.2.1's ESM build has a directory import (`export * from "../api"` with no `/index.js`) that Node's ESM resolver rejects, and for a 0.x range the caret pins the minor, so `^0.2.1` excludes the fix.

      Risks/Unknowns

      • `listIndexes()` now returns logical index names instead of every collection in the database. Anything treating it as "list collections" will see fewer entries.

      Acceptance Criteria

      Implementation Requirements

      • BYO collections are never created or dropped, and are read-only unless writes were explicitly opted into; the policy is persisted so it survives a restart and fails closed.
      • Managed-path behavior unchanged when `collectionName` is omitted.
      • `hybridQuery` version-gated at MongoDB >= 8.0.
      • Multimodal text serializes correctly; `baseUrl` optional and omitted when unset.

      Testing Requirements

      • Full package suite green: 217 vector tests plus 620 storage/filter tests.
      • Live Atlas 8.0.28 end-to-end, 23/23 checks: BYO lifecycle, write guard across a simulated restart, `textQuery`, `hybridQuery`, and `deleteIndex` preserving all operational documents.
      • VoyageAI unit tests on request shape and `baseUrl`; embedder verified against the live MongoDB-hosted endpoint.

      Documentation Requirements

      • Upstream docs updated: `reference/vectors/mongodb.mdx`, `rag/vector-databases.mdx`, `models/embeddings.mdx`, VoyageAI README, plus changesets.

      Follow Up Requirements

      • No release pending.
      • Optional, this repo: migrate `src/mastra/embed.ts` to `@mastra/voyageai` and evaluate `MongoDBVector.hybridQuery` in place of the hand-built pipelines in `src/retrieval/pipelines.ts`. Either way, correct the comment at `src/mastra/embed.ts:19-27`, which still lists the `baseUrl` gap and the `@huggingface/transformers` dependency as reasons to avoid the package. #19804 fixed the first and 0.4.0 dropped the second.

            Assignee:
            Unassigned
            Reporter:
            Mohammad Daoud Farooqi
            None
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: