Replace $lookup localField/foreignField $match placeholder index with an internal marker stage

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Integration
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Overview

      For a $lookup with a localField/foreignField join, the equality $match placeholder's position in the resolved foreign sub-pipeline is tracked as a raw index (_fieldMatchPipelineIdx, serialized as $_internalFieldMatchPipelineIdx). An index cannot survive desugaring (one stage expands to N stages) or view resolution, so the code compensates with stage-specific BSON pattern matching and shard-side relocation (relocateFieldMatchPlaceholder, computeDesugaredMongotFieldMatchIdx, computeHybridSearchFieldMatchIdx). Every new desugar variant breaks this and forces more pattern matching.

      Proposed change (POC)

      Replace the index with a self-identifying internal marker stage, $_internalLookupFieldMatch, that is a real element of the sub-pipeline, so its position is carried structurally by the pipeline itself:
      * New internal-only stage $_internalLookupFieldMatch with an empty spec; rejected from user requests; never executes ($lookup substitutes the concrete per-input-document $match into its slot, found once by name scan).
      * Gated on the existing gFeatureFlagExtensionsInsideHybridSearch (no new feature flag). Flag off: byte-for-byte legacy behavior.
      * Marker is serialized inside the sub-pipeline for remote dispatch; the shard finds it by scan; $_internalFieldMatchPipelineIdx is not written and relocation logic is dead code on the marker path.
      * Marker hidden from user-facing serialization (explain, query shape) and FLE2.

      Scope of Work

      * src/mongo/db/pipeline/document_source_internal_lookup_field_match.h/.cpp — new marker stage
      * src/mongo/db/exec/agg/internal_lookup_field_match_stage.cpp — never-execute exec mapping (tasserts if reached)
      * src/mongo/db/pipeline/document_source_lookup.cpp/.h — flag-gated marker emission at all placeholder sites; marker-aware insertion (including already-desugared sub-pipelines); scan replaces shard-side relocation; serialization changes
      * Unit tests in db_document_source_test; integration coverage via core, sharded passthrough, and mongot e2e suites

      Acceptance Criteria

      * Flag-on: marker replaces the empty $match placeholder everywhere; no wire index; no relocation; results identical to legacy for scalar/array/regex/missing local values
      * Flag-off: behavior and serialization unchanged
      * An unsubstituted marker reaching execution fails with a named tassert

            Assignee:
            Mariano Shaar
            Reporter:
            Mariano Shaar
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: