Summary:

Hybrid grep's $rankFusion fulltext branch has no per-branch limit and both branches post-filter after search, risking unbounded scans and under-returned scoped matches

Goal

Bound the fulltext branch of the hybrid grep pipeline the same way the vector branch already is, and make the path/glob scoping filter apply before result-set truncation rather than after, so scoped grep calls (path=, glob=) return complete results instead of silently dropping matches on large or narrowly-filtered collections.

Current state

verified on main, langchain_mongodb_deepagents_vfs/search.py

Item	Status
Vector branch of $rankFusion is bounded	vector_search_stage() (from pymongo_search_utils) bakes limit: top_k and numCandidates: top_k * oversampling_factor directly into the $vectorSearch stage itself, so its output is capped before any downstream filtering.
Fulltext branch of $rankFusion is bounded	_grep_hybrid() hand-rolls the fulltext sub-pipeline as \\{"$search": {...}}, *match instead of using the text_search_stage() helper already imported in this same file — which does accept a limit kwarg. No $limit stage exists inside the fulltext sub-pipeline. Only a single $limit: grep_limit exists, applied after the whole $rankFusion combination.
Path/glob scoping is pushed into the search stage	Both branches apply path/glob as a $match stage after the search/vectorSearch stage, not as part of the search query itself. Documented reason for the vector branch: $vectorSearch's native filter doesn't support $regex. No equivalent constraint applies to the fulltext branch — Atlas Search's own $search.compound.filter (or a text clause with a regex-capable filter) could carry source_path/filename scoping natively, but this is not currently attempted.
Non-hybrid fulltext-only fallback (embedding failure path)	Already correctly bounded — this path uses text_search_stage(..., limit=self._grep_limit, filter=pre_filter), so it isn't affected by this ticket.

Why this matters

• Unbounded scan risk: a common search term against a large collection lets the fulltext branch rank an unbounded number of candidate documents before the single trailing $limit ever trims it — cost scales with matches in the whole collection, not with grep_limit.
• Under-returned results on scoped queries: both branches fetch a small, globally-ranked candidate set (top grep_limit * 2 for vector; effectively unbounded-then-discarded for fulltext) and then discard anything outside the requested path/glob scope. If the scoped subset isn't well represented among the global top hits, grep(path=..., glob=...) can return fewer matches than actually exist, or none, even though real matches exist in MongoDB — this is a correctness gap, not just a latency one.
• MongoDB's own $rankFusion guidance recommends bounding each input pipeline, both for cost and because Reciprocal Rank Fusion's combination step is only meaningful over comparably-sized, bounded rankings per branch.

Steps

1. Add an explicit $limit inside the fulltext sub-pipeline of $rankFusion (e.g. grep_limit * 2, matching the vector branch's top_k), either by switching to the text_search_stage() helper or adding the stage inline.
2. Investigate pushing path/glob filtering into the fulltext branch's $search stage itself (via compound.filter with a regex or wildcard operator on indexed source_path/filename), rather than a post-search $match, so scoping doesn't compete with relevance ranking for the same fixed candidate budget.
3. For the vector branch, evaluate whether oversampling_factor needs to scale with filter selectivity (e.g. increase candidates when path/glob is set) as a mitigation if native pre-filter pushdown isn't feasible for vector search.
4. Add regression tests that construct a collection where the scoped subset is deliberately not among the global top-N by relevance/similarity, and assert grep(path=..., glob=...) still returns the in-scope matches.

Definition of Done

• The fulltext sub-pipeline inside _grep_hybrid()'s $rankFusion has its own explicit $limit, verified by reading the constructed pipeline in a unit test (not just relying on the final outer $limit).
• A test demonstrates that a grep() call scoped by path or glob to a subset of documents that would not appear in an unscoped top-N still returns those matches (i.e., scoping is no longer just a post-filter on a fixed, unrelated global candidate set).
• No regression in existing grep/search.py unit and integration tests.
• just lint and just typing pass.
• Decision needed: whether native filter pushdown into $search.compound.filter is pursued now, or deferred with only the $limit fix + oversampling mitigation landing in this ticket (see Steps #2–3).

Decisions needed

1. Pushdown vs. limit-only fix. Pushing path/glob into $search.compound.filter directly addresses the under-return risk more completely than raising limits/oversampling, but is a larger change and needs verification that Atlas Search's compound.filter + regex/wildcard operators support the same prefix/glob semantics this package currently gets from client-side $match. Options: (a) full pushdown now; (b) limit fix + oversampling mitigation now, pushdown as a follow-up ticket.

Notes

• This does not affect the plain full-text-only fallback path (used when query embedding fails) — that one already uses text_search_stage() with an explicit limit and is unaffected.
• Related, out of scope for this ticket: the non-Atlas _grep_regex() fallback performs an unindexed content regex scan bounded only by a 5s max_time_ms server-side timeout — tracked separately.