Fix double view application in AggExState and guard against it in LiteParsedNestedPipelines

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Fixed
    • Priority: Major - P3
    • 9.0.0-rc1
    • Affects Version/s: None
    • Component/s: Catalog
    • None
    • Query Integration
    • Fully Compatible
    • QI 2025-02-17
    • 200
    • None
    • None
    • None
    • None
    • None
    • None
    • None

        1. Symptom

      On a sharded cluster, an aggregation over a chain of views involving a bare `$lookup`
      (`localField`/`foreignField`, no user `pipeline:`) could crash a shard with:

      ```
      tassert 12792401: expected an empty subpipeline list when materializing a view subpipeline
      src/mongo/db/pipeline/lite_parsed_document_source_nested_pipelines.h
      LiteParsedDocumentSourceNestedPipelines<LiteParsedLookUp>::materializeViewSubpipeline()
      ```

      First seen in `concurrency_sharded_replication_last_lts_new_old_old_new`
      (`view_catalog_cycle_lookup.js`).

        1. Root cause: the binding loop runs twice on the same stage

      View-subpipeline materialization for a bare `$lookup`/`$unionWith`/`$graphLookup` over a view
      happens in one place — the binding loop inside `AggCatalogState::extendResolvedNamespaces`:

      ```cpp
      for (auto& [_, entry] : resolvedNamespaces) {
      if (auto* parsed = entry.getMutableParsedPipeline())

      { PipelineResolver::resolveInvolvedNamespacesOnLiteParsedPipeline( &parsed->pipeline(), entry.getResolvedNamespace(), resolvedNamespaces); }

      }
      ```

      For each involved view, this calls `handleView → bindResolvedNamespace` on every stage. A bare
      `$lookup` over a view (`_noUserPipeline == true`) responds by calling `materializeViewSubpipeline`,
      which folds the view's pipeline into the stage's `_pipelines` (going from empty to size 1).
      `materializeViewSubpipeline` asserts `_pipelines` is empty on entry — it assumes it runs *once per*
      *stage object*.

      During a single from-router aggregate execution on a shard, this loop runs in *two* resolution
      passes:

          1. Pass 1 — memoized resolution

      `AggCatalogState::getResolvedInvolvedNamespaces()` is memoized. Its first call runs
      `resolveInvolvedNamespaces → extendResolvedNamespaces →` the binding loop, materializing the bare
      `$lookup`'s view subpipeline into the *memoized* map's entry. That entry's stage now has
      `_pipelines.size() == 1`.

          1. Pass 2 — proactive router resolution

      `AggCatalogState::maybeProactivelyResolveInvolvedNamespaces()` runs on a from-router shard (or the
      merging half of a sharded aggregation) and does two relevant things:

      ```cpp
      auto resolvedMap = getResolvedInvolvedNamespaces(opCtx); // (A) COPY of the memoized map
      ...
      if (mainIsView) {
      ...
      extendResolvedNamespaces(opCtx, namespaces, resolvedMap); // (B) runs the binding loop AGAIN
      }
      ```

      • *(A)* `getResolvedInvolvedNamespaces()` returns a reference to the already-materialized memoized
        map; `auto` makes `resolvedMap` a *copy*. The copy deep-clones each `ResolvedNamespace`, and
        `LiteParsedPipeline`'s copy constructor clones each stage via `stage->clone()`, which copies
        `_pipelines`. So the copied `$lookup` stage *already carries the materialized subpipeline*
        (`_pipelines.size() == 1`), while `_noUserPipeline` is still `true`.
      • *(B)* `extendResolvedNamespaces` runs the same binding loop over that copy. It re-binds the
        cloned `$lookup`, which already has a non-empty `_pipelines`, so `materializeViewSubpipeline` is
        entered a second time and trips the `_pipelines.empty()` invariant → `tassert 12792401`.

      So the "double call" is the binding loop in *pass 1 (memoized map)* and again in *pass 2 (a copy*
      *that inherited pass 1's materialization)*.

        1. Why it only reproduced under multiversion + concurrency

      Pass 2's re-binding is gated by two conditions:

      1. *`mainIsView == true`* — the second `extendResolvedNamespaces` is inside `if (mainIsView)`, so
      the shard's main namespace must itself be a view. In the FSM workload a concurrent
      `remapViewToView` / `remapViewToCollection` makes a namespace the router treated as a collection
      be a *view* by the time the shard (or the kickback retry) processes it.
      2. *An older (last-LTS) router* — a latest-binary router pre-resolves foreign views for the shard,
      so the shard skips this two-pass resolution. An older router does not, so the latest-binary shard
      resolves involved namespaces itself and reaches the proactive pass.

      A single-version cluster, or a static (non-concurrent) query, does not line up both conditions in
      one execution, which is why the crash only surfaced in the mixed-version concurrency suite.

        1. Fix

      Make the second pass not re-materialize an already-materialized stage. Two layers:

      • *Structural:* `extendResolvedNamespaces` binds only the namespaces it newly added in this
        call, rather than re-binding every entry in the (possibly pre-materialized) map. A pre-existing
        entry can only reference namespaces that were also present before this call (view resolution is
        transitive), so it never needs re-binding.
      • *Defensive (idempotency):* `LiteParsedDocumentSourceNestedPipelines` records
        `_viewSubpipelineMaterialized` once `materializeViewSubpipeline` runs. The flag is copied
        memberwise with `_pipelines` when the stage is cloned, so a re-bind of a copied stage is a no-op
        instead of tripping the invariant.

      Within one aggregate execution the resolved-namespace map is built from a single consistent catalog
      snapshot, so skipping the redundant second materialization yields the identical view pipeline — no
      view is lost, applied twice, or applied stale.

            Assignee:
            Finley Lau
            Reporter:
            Finley Lau
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: