Summary

In sharded topologies, the IFR kickback (@throwIfrKickbackIfNecessary@) causes a shard to hit a tripwire assertion (tassert) when retrying a @$scoreFusion@/@$rankFusion@ pipeline containing extension @$vectorSearch@ with the legacy @$vectorSearch@. This crashes the mongod process serving that shard.

This was discovered while re-enabling tests excluded for SERVER-130918: after that fix, 14 of the 16 previously-excluded with_mongot_extension_* hybridSearch tests now pass, but 2 still fail – only in sharded suites.

Environment

• Suites: with_mongot_extension_sharded_cluster, {
  Unknown macro: {with_mongot_extension_sharded_collections}

  }

• Passes on with_mongot_extension_single_node – this is sharded-topology specific.
• featureFlagExtensionsInsideHybridSearch: false (the mongot-extension provides @$vectorSearch@; IFR kickback is expected to retry with the legacy implementation)

Failure

Both affected tests hit the same tassert on a shard:

Location10535800: The metadata dependency tracker determined $scoreFusion input pipeline does not
generate score metadata, despite the input pipeline stages being previously validated as such.

at src/mongo/db/pipeline/document_source_score_fusion.cpp:83, inside parseScoredSelectionPipelines. Because this is a tassert (tripwire assertion), it kills the mongod process on that shard rather than just failing the command.

jstests/with_mongot/e2e/hybridSearch/hybrid_search_weights_test.js

Repro:

python3-venv/bin/python buildscripts/resmoke.py run --suites=with_mongot_extension_sharded_cluster \
--force-excluded-tests \
--additionalFeatureFlags=featureFlagRankFusionFull,featureFlagSearchHybridScoringFull \
jstests/with_mongot/e2e/hybridSearch/hybrid_search_weights_test.js

Also reproduces on with_mongot_extension_sharded_collections.

The test's runTestExpectError case at hybrid_search_weights_test.js:100 issues a @$rankFusion@ expected to fail client-side with a weights-count validation error; instead shard s1 hits the tassert above and the mongod on that shard crashes.

jstests/with_mongot/e2e/hybridSearch/score_fusion_lookup.js

Repro:

python3-venv/bin/python buildscripts/resmoke.py run --suites=with_mongot_extension_sharded_cluster \
--force-excluded-tests \
--additionalFeatureFlags=featureFlagSearchHybridScoringFull \
jstests/with_mongot/e2e/hybridSearch/score_fusion_lookup.js

Also reproduces on with_mongot_extension_sharded_collections.

The test issues a @$lookup@ subpipeline containing @$scoreFusion@ over extension @$vectorSearch@, expecting it to fail with code 51047 (extension-in-lookup rejection, see SERVER-131118). Instead it fails with the tassert above.

Initial analysis

Both failures involve @$scoreFusion@/@$rankFusion@ containing extension @$vectorSearch@, running on a sharded topology, where the IFR kickback causes runAggregate to retry the command with featureFlagVectorSearchExtension disabled per-operation (retryOnWithStateMultiLoop in ClusterAggregate::runAggregate). On a single node this retry produces a consistent re-parse. In the sharded case, the retry appears to happen inconsistently across mongos and the individual shards: the metadata-dependency tracker on at least one shard (s1 in hybrid_search_weights_test.js, s0 in score_fusion_lookup.js) ends up validating the input pipeline as producing score metadata, but then determines at execution time that it does not – i.e. the shard is retrying with a different view of whether the @$vectorSearch@ stage is the extension or legacy version than what the lite-parse validation assumed.

This is consistent with the router-side retry being properly scoped to disable the extension flag, while something in the shard-side command dispatch or re-parse does not fully propagate the same flag state, or the lite-parsed pipeline built during validation is cached/reused across the retry boundary.

This looks like a distinct bug from the original SERVER-130918 report (which was about the kickback error reaching the client at all) – here the kickback does retry, but sharded re-parsing of the retried command is inconsistent.

Next steps

• Investigate ClusterAggregate::runAggregate's retry loop and how LiteParsedScoreFusion/LiteParsedRankFusion validation state carries across the per-shard retry in a sharded cluster.
• Both tests remain excluded in buildscripts/resmokeconfig/matrix_suites/overrides/with_mongot_extension.yml pending this fix.