Investigate behavior of setting metadata for input pipelines in sharded query

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Integration
    • None
    • 3
    • TBD
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      In SERVER-104472, a bug was discovered for $scoreFusion queries where the normalization is 'minMaxScaler' and at least one of the input pipelines is a $score stage with scoreDetails set to true that computes a score like this:

      $score: {            score: {$add: [10, 2]},            normalizeFunction: "none",            scoreDetails: scorePipeline2ScoreDetails        } 

      Because 'minMaxScaler' normalization inserts an internalSetWindowsField stage in the pipeline, that stage and any subsequent stages are run on the merging node instead of parallel on the shards. Thus, the $addFields stages for the input pipeline's rawScore and scoreDetails fields (the latter of which references $meta: scoreDetails) were executed on the merging node because they showed up after internalSetWindowsField in the desugared pipeline.

      With this desugared pipeline order, the input pipeline's scoreDetails (which should have been saved under the field <inputPipelineName>_scoreDetails) never bubbled up to the final scoreDetails object (was completely omitted). To fix this behavior, both $addFields stages were inserted before the internalSetWindowsFields stage, guaranteeing that they'd be run on the shards (before the pipeline split). 

      In theory, though, each input pipeline's $setMetadata stages (for score and scoreDetails), which are run on the shards, should individually set each document's metadata. Thus, this metadata should be able to be referenced by stages executed on the merging node. The bug mentioned above seems to suggest otherwise. More investigation needs to be done into why the original pipeline failed for sharded test cases.

              Assignee:
              Unassigned
              Reporter:
              Adithi Raghavan
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: