Bring $rankFusion desugaring in-line with $scoreFusion (use single 'internal_fields' object)

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Unresolved
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Query Integration
    • None
    • 3
    • TBD
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      During SERVER-104736 (PR) we refactored the $scoreFusion desugar to use a single 'internal_fields' object to always kept all internal fields used in the desugar in a single top level object. This was originally done to support being able to reference collection level fields in your combination method expression, but also in general creates a more simplified / predictable intermediate document structure that always looks like:

      {_id: ..., _internal_scoreFusion_docs: {...}, _internal_scoreFusion_internal_fields: {...}}

       

      The purpose of this ticket is to make $rankFusion's intermediate document structure look the same, and conceptually align the desugars of both stages as much as possible.

       

      This has 2 immediate benefits we can identify:

      • query stats: right now query stats are logged after a pipeline is fully desugared and optimized. So for a hybrid search it will be difficult to discern that it was a hybrid search, without these "internal_field" prefixes which clearly label the pipeline is a hybrid search
      • more consistent behavior between stages, especially in sharded clusters: there are multiple desugared pipelines we could write that conceptually arrive at the same result. In theory, they should all behave the same way (after re-writes like optimization) but in practice there can be peculiarities of how specific pipelines execute, especially when sharding is involved. By making the stages desugar in a similar way, we will minimize the inconsistencies in their behavior.
        • One concrete example of this is when testing upgrade / downgrade behavior, we noticed a difference between $rankFusion / $scoreFusion (rank fusion did something non-ideal in mixed version clusters, whereas score fusion behaved as we would want it to). See if after this desugar is changed if SERVER-108470 becomes effectively resolved and 'whenBinariesAreLatestAndFCVIsLastLTS' can be set to the function where rank fusion is always expected to fail.

              Assignee:
              Adithi Raghavan
              Reporter:
              Joe Shalabi
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: