TL;DR

Add MongoDBVector.hybridQuery() with separate vector, text, and optional reranking configurations. Support caller-generated and automatic embeddings, expose independent retrieval limits, and default native reranking to rerank-3. Leave existing search APIs unchanged.

Context

Mastra's MongoDB integration supports searching by meaning through vector search and by keywords through full-text search. Hybrid search combines these results into one ranked list using MongoDB's $rankFusion stage.

Reranking evaluates the retrieved documents against a query to refine their order. Today, hybridQuery() does not expose native reranking, so applications must configure and call a reranker separately.

MongoDB supports $rerank after $rankFusion, allowing retrieval, fusion, and reranking in one aggregation pipeline. Introduce a new method with explicit configuration for each operation without changing existing APIs or requiring users to replace their embedding provider.

Desired User Experience

Examples assume an initialized store, registered search indexes, a query embedding, and document text stored in metadata.text.

Before: configure and call a reranker separately

import { rerankWithScorer } from '@mastra/rag';
import { createVoyageReranker } from '@mastra/voyageai';
// Requires a separate Voyage client configuration and VOYAGE_API_KEY.
const scorer = createVoyageReranker('rerank-3');
const results = await store.hybridQuery({
  indexName: 'articles', 
  queryVector: embedding, 
  query: question, 
  paths: ['metadata.text'], 
  topK: 50,
});
const reranked = await rerankWithScorer({ results, query: question, scorer, options: { topK: 5 } });

The Before example uses a model supported by the current Mastra Voyage helper.

After: configure retrieval and native reranking together

Proposed API; not currently supported.

const results = await store.hybridQuery({
  indexName: 'articles',
  vector: { queryVector: embedding, limit: 50, numCandidates: 1000 },
  text: { query: question, paths: ['metadata.text'], limit: 50 },
  rerank: { query: question, paths: ['metadata.text'], model: 'rerank-3', numDocsToRerank: 50 },
  topK: 5, // Return up to five results after reranking the fused candidates.
});
// No application-side reranker, separate Voyage API key, or result mapping.

Hybrid retrieval without reranking

const results = await store.hybridQuery({
  indexName: 'articles_auto', // Automatic-embedding index over the document field.
  vector: { query: question, limit: 50 }, // MongoDB embeds the query text.
  text: { query: question, paths: ['document'], limit: 50 },
  topK: 5,
  // rerank: {}, // No rerank configuration: return the fused vector and keyword results.
});

How the search options are organized

A hybrid search has two retrieval branches: vector finds documents by meaning, and text finds keyword matches in selected fields. Each contributes candidates to a combined ranking. Optional rerank configuration refines that ranking before the final result limit.

The complete proposed parameter shape is:

type HybridQueryParams = {
  indexName: string; 
  topK?: number; 
  filter?: MongoDBVectorFilter;
  weights?: {
    vector?: number; 
    text?: number 
  }; 
  metadataMode?: 'field' | 'document';
  vector: 
  // Manual embedding vs Autoembedding mode
  ({ queryVector: number[] } | { query: string; model?: string })
  &
  { 
    numCandidates?: number; 
    limit?: number 
  };
  text: { 
    query: string; 
    paths: string[]; 
    searchIndexName?: string; 
    limit?: number 
  };
  rerank?: { 
    query: string; 
    paths: string[]; 
    model?: string; 
    numDocsToRerank: number 
  };
};

Compared with today's hybridQuery(), the new hybridQuery() organizes parameters as follows. The existing method remains unchanged.

Task

Goal: Let applications control hybrid retrieval and optionally rerank results in one search call without changing embedding providers or breaking existing integrations.

Where: MongoDBVector implementation, with focused coverage in the MongoDB vector tests.

Product Acceptance Criteria

• [ ] The Desired User Experience works on a supporting Atlas deployment with both managed and bring-your-own collections (BYOC) (NODE-7739).
• [ ] Existing query(), textQuery(), and hybridQuery() signatures and behavior remain unchanged.
• [ ] Invalid requests and unavailable reranking produce clear errors with preserved causes and no silent fallback.
• [ ] Parameters are clearly documented with JSDoc comments.
• [ ]Sufficient test coverage has been added to support the described experience.
• [ ] A DOCSP documentation ticket exists.

Out of Scope

• Changes to existing search APIs.
• Arbitrary aggregation pipelines, caller-defined projections, or additional search operators.
• Multiple vector/text branches, per-branch filters, or new vector-path/index overrides.

Additional Context

Model and deployment compatibility

The current MongoDB $rerank reference requires MongoDB 9.0+ with Native Reranking enabled in Atlas project settings. Native rerank-3 additionally requires an M10-or-higher dedicated cluster and is currently Preview.

Self-managed and Atlas Local deployments are excluded. MongoDB caps numDocsToRerank at 1000, and missing reranking fields produce server errors. Preserve these errors and their causes; do not silently substitute another model or skip reranking.

Appendix: TypeScript and equivalent MQL

Proposed TypeScript call

const results = await store.hybridQuery({
  indexName: 'articles',
  vector: { queryVector: embedding, limit: 50, numCandidates: 1000 },
  text: { query: question, paths: ['metadata.text'], limit: 50 },
  rerank: { query: question, paths: ['metadata.text'], model: 'rerank-3', numDocsToRerank: 50 },
  topK: 5, // Return up to five results after reranking the fused candidates.
});
// No application-side reranker, separate Voyage API key, or result mapping.

Equivalent MongoDB Query Language (MQL)

Here, db.articles is the resolved collection, vectorIndexName and textIndexName are the resolved physical search index names, and embeddingPath is the configured embedding field. The final projection uses the default metadata and document fields.

db.articles.aggregate([
  // Retrieve candidates from both branches and fuse with equal default weights.
  {
    $rankFusion: {
      input: {
        pipelines: {
          vector: [
            {
              $vectorSearch: {
                index: vectorIndexName,
                path: embeddingPath,
                queryVector: embedding,
                numCandidates: 1000,
                limit: 50,
              },
            },
          ],
          text: [
            {
              $search: {
                index: textIndexName,
                text: {
                  query: question,
                  path: ['metadata.text'],
                },
              },
            },
            { $limit: 50 },
          ],
        },
      },
    },
  },

  // Rerank up to 50 fused source documents.
  {
    $rerank: {
      query: { text: question },
      path: ['metadata.text'],
      model: 'rerank-3',
      numDocsToRerank: 50,
    },
  },

  // Apply topK after reranking, not before.
  { $limit: 5 },

  // Return the selected fields and final reranking score.
  {
    $project: {
      _id: 1,
      metadata: '$metadata',
      document: '$document',
      score: { $meta: 'score' },
    },
  },
]);