Rewrite `DocumentSourceLookUp::serializeToArray` to consume the LiteParsedPipeline for view-resolved sub-pipeline serialization

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Gone away
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Integration
    • ALL
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Summary

      `DocumentSourceLookUp::serializeToArray`, when serializing for remote dispatch against a
      view-resolved foreign namespace, currently prepends raw BSON read directly from
      `_sharedState->resolvedPipeline` (the resolver's raw output captured at DS construction time).
      This bypasses the LiteParsedPipeline entirely, so any view-rewriting work done at LP-time
      (e.g. `LiteParsedLookUp::bindViewInfo`, `LiteParsedGraphLookUp::bindViewInfo`,
      `LiteParsedPipeline::bindViewInfoToStages`) does not influence the wire payload for the
      prepended view-pipeline slice. As a result, when a view's definition contains nested
      `$lookup` / `$graphLookup` / `$unionWith` referencing other views, those nested view names
      leak onto the wire on retry after a kickback, causing the shard to re-kick-back with the
      same closure mongos already has, and the router's convergence guard
      (`uassert 12451400`) fires.

      Current state (after SERVER-121972)

      There are two parallel channels feeding the wire payload for `$lookup` against a view:

      1. {}LP channel{} — `LookUpStageParams.liteParsedPipeline` → introspection pipeline →
      DS stages → serialized via the normal DS serialization at the tail of
      `serializeToArray`. View resolution applied here via `_resolvedSubPipelineView`.
      2. {}Raw-BSON channel{} — `_sharedState->resolvedPipeline` (a `vector<BSONObj>` captured
      from the resolver) → prepended verbatim at
      `document_source_lookup.cpp:~1153`. Untouched by LP.

      For bare-target `$lookup` (no user-supplied `pipeline:` field) against a view, the view's
      own stages live in the raw-BSON channel — they are exactly the slice that the L1153 prepend
      emits. The LP channel cannot rewrite this because it isn't the source of those bytes.

      SERVER-121972 ships a defensive band-aid on the DS side that rewrites the prepended slice
      in place using `view_dispatch_bson_helpers::recursivelyRewriteSubPipeline` whenever the
      foreign was a view, the request is for remote dispatch, the stage was a bare-target
      `$lookup`, the field-match pipeline index is positive, and the serialization is not for
      query stats / explain / FLE2. This works but duplicates the rewrite logic that LP already
      implements and reaches across the LP/DS boundary.

      Proposed change

      Drop the raw-BSON prepend at L1153 in `DocumentSourceLookUp::serializeToArray` and instead
      serialize the relevant slice of `liteParsedPipeline` (which already carries the view
      rewrites via `_resolvedSubPipelineView` after `bindViewInfo`). The DS should be a strict
      consumer of LP-resolved data for view-resolved sub-pipeline serialization on the wire path.

      Concretely:

      • Replace the `_sharedState->resolvedPipeline.begin() .. + *_fieldMatchPipelineIdx`
        insert with a call that serializes the LP's sub-pipeline (the same `[0,
        internalFieldMatchPipelineIdx)` slice) and inserts those `BSONObj`s instead.
      • Verify the index semantics line up between the raw-BSON channel and the LP channel
        (both should describe "stages before the synthesized `$match` placeholder"; if not,
        introduce a translation step in `LookUpStageParams` so they share a single source of
        truth).
      • Once the DS no longer reads from `_sharedState->resolvedPipeline` for remote
        serialization, audit whether `_sharedState->resolvedPipeline` is still needed for any
        other code path; if not, remove it.
      • Delete the band-aid in `DocumentSourceLookUp::serializeToArray` introduced by
        SERVER-121972 and the supporting helpers in
        `src/mongo/db/pipeline/view_dispatch_bson_helpers. {h,cpp}
        ` if no other caller remains.

      Why this matters beyond layering

      The current band-aid only kicks in under a narrow predicate set (bare-target, positive
      field-match index, not query stats / explain / FLE2). Any future code path that ends up
      prepending raw resolved BSON for remote dispatch will re-introduce the same view-name
      leak unless it remembers to call the same helper. Centralizing the rewrite in the LP
      channel makes "DS serialization consumes LP-resolved data" the invariant, removing the
      class of bug rather than patching individual call sites.

      Acceptance criteria

      • `DocumentSourceLookUp::serializeToArray` does not read raw BSON from
        `_sharedState->resolvedPipeline` for any remote-dispatch code path.
      • `view_catalog_cycle_lookup.js` FSM workload in
        `concurrency_sharded_replication_with_balancer_and_config_transitions` runs without
        triggering `uassert 12451400` (convergence guard) at all — not just tolerating it via
        `commandWorkedOrFailedWithCode`.
      • Existing tests for `$lookup` against views (sharded and unsharded), including
        `jstests/sharding/query/lookup/*` and the relevant FSM workloads, continue to pass.
      • The SERVER-121972 band-aid in `document_source_lookup.cpp` is removed.

      Background reading

      • SERVER-121972 PR — for the LP-side `bindViewInfo` work (LiteParsedLookUp,
        LiteParsedGraphLookUp) that this followup builds on.
      • `src/mongo/db/pipeline/view_dispatch_bson_helpers.h` — the in-place BSON rewriter that
        this followup is intended to make unnecessary on the DS side.
      • The convergence guard at `cluster_aggregate.cpp` (uassert 12451400) — the symptom that
        surfaces when the wire payload still leaks view names after a kickback.

       

      We can delete getShardDispatchBson() when this is complete

            Assignee:
            Finley Lau
            Reporter:
            Finley Lau
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: