Description

On a sharded cluster, an extension $search inside a $lookup subpipeline that also uses a localField/foreignField join fails with:

Location40602: $_internalDocumentResultsAndMetadata is only valid as the first stage in a pipeline

No view is involved. When the $lookup has localField/foreignField, an equality $match is injected at the front of the subpipeline to implement the join. When the user's subpipeline begins with an extension $search (which desugars to $_internalDocumentResultsAndMetadata wrapping $_extensionSearch), that injected prefix pushes $_internalDocumentResultsAndMetadata out of first position, tripping the first-stage-only check (Location40602).

This is distinct from SERVER-131212, which is the view case in computeDesugaredMongotFieldMatchIdx and surfaces as Location12761201 ("internalFieldMatchPipelineIdx out of range of resolvedPipeline"). This ticket is the no-view, stage-position variant (Location40602).

It is also distinct from SERVER-131308, which fixes the view-resolution ordering for an extension $vectorSearch/$search in a $lookup on a view. That fix does not touch the localField/foreignField equality-match prepend path, so this scenario remains broken.

Feature flag configuration

Reproduces with the extension flags where $search runs through the Extensions API:

• {
  Unknown macro: {featureFlagSearchExtension}

  }

• {
  Unknown macro: {featureFlagExtensionsInsideHybridSearch}

  }

• {
  Unknown macro: {featureFlagVectorSearchExtension}

  }

(i.e. the with_mongot_extension_vectorSearch_disabled_* sharded suites.)

Failing query

db.outerColl.aggregate([
{
$lookup: {
from: "innerColl",
localField: "state",
foreignField: "state",
pipeline: [
{ $search: searchStageDefinition }, // extension $search, returnStoredSource: true
{ $project: \{ _id: 0 } }
],
as: "matchedDocs"
}
}
]);

Shard-targeted pipeline (as sent to the shard, showing the desugared extension prefix):

[
{ $lookup: {
from: "innerColl", as: "matchedDocs",
localField: "state", foreignField: "state",
pipeline: [
{ $_internalDocumentResultsAndMetadata: \{ source: { $_extensionSearch: { ... } }, ... } },
{ $replaceRoot: \{ newRoot: "$storedSource" } },
{ $project: \{ _id: false } }
]
} }
]

Reproduction

jstests/with_mongot/e2e/sharding_no_passthrough/subpipeline_storedSource_chunk_migration.js

Run under any sharded with_mongot_extension_vectorSearch_disabled_* suite, e.g.:

python3 buildscripts/resmoke.py run \
--suites=with_mongot_extension_vectorSearch_disabled_sharded_cluster \
jstests/with_mongot/e2e/sharding_no_passthrough/subpipeline_storedSource_chunk_migration.js

Observed error

Location40602: Executor error during aggregate command on namespace:
subpipeline_storedSource_chunk_migration.outerColl ::
caused by :: $_internalDocumentResultsAndMetadata is only valid as the first stage in a pipeline

Workaround

The test is currently excluded in the sharded vectorSearch_disabled mappings via
exclude_extension_search_on_sharded_view_incompatible in
buildscripts/resmokeconfig/matrix_suites/overrides/with_mongot_extension.yml
(TODO tagged to this ticket). Remove that exclusion when this is fixed.

Suggested fix direction

The localField/foreignField equality-$match injection for the $lookup subpipeline
should account for an extension-desugared leading stage
($_internalDocumentResultsAndMetadata/$_extensionSearch) the same way view application now
does (see SERVER-131308), so the extension stage remains first (or the join match is applied without
displacing it).