Add sort_key metadata to LogicalStage, and add it to returned documents during execution.

XMLWordPrintableJSON

    • Query Integration
    • None
    • 3
    • TBD
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      During the execution phase of the query, reporting of available metadata in a document is required. For the most part, this can be handled by Document::fromBSONWithMetadata() , which allows us to lazily evaluate the underlying document on demand for metadata fields which may be found in the document.

      In some cases, we may want to add a SortKey metadata to the documents during execution. The SortKey is a special case, as it may require the extension to be aware of SortPattern and expression $meta.

       

      The Extensions API will add get_sort_key_metadata to the MongoExtensionLogicalAggregationStage vtable, in order for an extension to report any SortKey metadata which must be added to the document by the execution of the stage. 

       

      SortKey metadata can be broken down into two scenarios:

      1. Simple: In this case, the SortKey is a single value, typically derived from the document being processed in the pipeline. This is relevant for $vectorSearch, where the SortKey is the value of $vectorSearchScore.
      2. Complex: In this case, the sort key is specified by a sort pattern specification provided by the user stage definition. This sort pattern is then used to generate a sort key in the Host. This is relevant for $search.

       

       

       

      The extension is responsible for serializing a BSON object with the following format to report the sort key metadata. 

       

       

      { 
        sortKey: <field name in document to be used as sort key>, 
        sortPattern: [array of sort specifications (i.e $meta)] 
      }

       

       

      An extension may only report either a “sortPattern” or a “sortKey” field. In this project, we will only support an extension specifying a fieldname in the document as the “sortKey” field. 

      At instantiation time, DocumentSourceExtension will read either the “sortKey” or “sortPattern”, to be used later during the execution phase to populate the SortKey metadata on the documents.

      When a stage undergoes execution, if a stage reports a metadata sort key, we will need to update the document with the sort key:

          tassert(7828500,            "Expected vector search distance to be present",            output.metadata().hasVectorSearchScore());
      
      
      // We actually don't have the logical stage at this time, so we'll need to already have read the sort key metadata earlier and pass it to the executable stage.
      // We should ensure that we only support a sortKey for the key metadata.
      // we probably want to have some abstraction that stores the sort key metadata.
      auto sortKeyMetadata = logicalStage.get_sort_key_metadata();
      if (sortKeyMetadata) {
         output.metadata().setSortKey(Value{output[sortKeyMetadata]}, true /*isSingleElementKey*/);
      }

       

       

            Assignee:
            Unassigned
            Reporter:
            Santiago Roche
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: