-
Type:
Task
-
Resolution: Unresolved
-
Priority:
Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
None
-
Query Integration
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Overview
For a $lookup with a localField/foreignField join, the equality $match placeholder's position in the resolved foreign sub-pipeline is tracked as a raw index (_fieldMatchPipelineIdx, serialized as $_internalFieldMatchPipelineIdx). An index cannot survive desugaring (one stage expands to N stages) or view resolution, so the code compensates with stage-specific BSON pattern matching and shard-side relocation (relocateFieldMatchPlaceholder, computeDesugaredMongotFieldMatchIdx, computeHybridSearchFieldMatchIdx). Every new desugar variant breaks this and forces more pattern matching.
Proposed change (POC)
Replace the index with a self-identifying internal marker stage, $_internalLookupFieldMatch, that is a real element of the sub-pipeline, so its position is carried structurally by the pipeline itself:
* New internal-only stage $_internalLookupFieldMatch with an empty spec; rejected from user requests; never executes ($lookup substitutes the concrete per-input-document $match into its slot, found once by name scan).
* Gated on the existing gFeatureFlagExtensionsInsideHybridSearch (no new feature flag). Flag off: byte-for-byte legacy behavior.
* Marker is serialized inside the sub-pipeline for remote dispatch; the shard finds it by scan; $_internalFieldMatchPipelineIdx is not written and relocation logic is dead code on the marker path.
* Marker hidden from user-facing serialization (explain, query shape) and FLE2.
Scope of Work
* src/mongo/db/pipeline/document_source_internal_lookup_field_match.h/.cpp — new marker stage
* src/mongo/db/exec/agg/internal_lookup_field_match_stage.cpp — never-execute exec mapping (tasserts if reached)
* src/mongo/db/pipeline/document_source_lookup.cpp/.h — flag-gated marker emission at all placeholder sites; marker-aware insertion (including already-desugared sub-pipelines); scan replaces shard-side relocation; serialization changes
* Unit tests in db_document_source_test; integration coverage via core, sharded passthrough, and mongot e2e suites
Acceptance Criteria
* Flag-on: marker replaces the empty $match placeholder everywhere; no wire index; no relocation; results identical to legacy for scalar/array/regex/missing local values
* Flag-off: behavior and serialization unchanged
* An unsubstituted marker reaching execution fails with a named tassert