Add $score with normalization inside $scoreFusion tests

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Query Integration
    • None
    • 3
    • None
    • None
    • None
    • None
    • None
    • None

      Currently we have fully implemented all normalization options for both $score and $scoreFusion. We also have tests for $score inside input pipelines in $scoreFusion.

       

      We have not, however, tested putting a $score inside a $scoreFusion with different normalization options for $score ($sigmoid and $minMaxScaler); all of them currently have $score's normalization as "none".

       

      Furthermore, we should add these test with different normalization options to the outer $scoreFusion as well.

       

      So the different combinations we should consider are:

      $scoreFusion: {
        pipelines: {
          p1: {$score: {score: ..., normalizeFunction: "sigmoid"}},
          p2: ...
        }
        normalization: "none"
      }
      $scoreFusion: {
        pipelines: {
          p1: {$score: {score: ..., normalizeFunction: "minMaxScaler"}},
          p2: ...
        }
        normalization: "none"
      } 
      $scoreFusion: {
        pipelines: {
          p1: {$score: {score: ..., normalizeFunction: "sigmoid"}},
          p2: ...
        }
        normalization: "sigmoid"
      }
      $scoreFusion: {
        pipelines: {
          p1: {$score: {score: ..., normalizeFunction: "minMaxScaler"}},
          p2: ...
        }
        normalization: "sigmoid"
      } 
      $scoreFusion: {
        pipelines: {
          p1: {$score: {score: ..., normalizeFunction: "sigmoid"}},
          p2: ...
        }
        normalization: "minMaxScaler"
      } 
      $scoreFusion: {
        pipelines: {
          p1: {$score: {score: ..., normalizeFunction: "minMaxScaler"}},
          p2: ...
        }
        normalization: "minMaxScaler"
      } 

      We should also set the scoreDetails to true for a some of these combinations and add them to score_fusion_score_details_test.js

       

      Note: In SERVER-104742 we found that there were issues in sharded clusters with how the pipeline was split with respect to $_internalSetWindowFields (this is inserted with $minMaxScaler normalization on $scoreFusion). The solution was to push some $addFields stages (for rawScore and scoreDetails) before the $_internalSetWindowFields. We should watch out for the same situation when $minMaxScaler normalization is set in $score. 

            Assignee:
            Unassigned
            Reporter:
            Joe Shalabi
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: