$idLookup could be responsible for apply limit to DistributedPlanLogic

XMLWordPrintableJSON

    • Query Integration
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      When a stage desugars with an $idLookup, it can choose to pass a limit value to be applied to the document stream. That limit only applies per-node though, so a sharded $idLookup could return N * limit documents overall, where N is the number of data-bearing nodes involved in the query.

      Today, $vectorSearch and $search (which desugar with an $idLookup node) compensate by being responsible themselves for adding a $limit to the DistributedPlanLogic merging pipeline. We could consider having $idLookup provide its own DPL to apply the limit optimization so that stages that desugar with $idLookup and want to apply a limit do not need to worry about the sharded implications of that limit.

      To be clear, this is not required for any pipelines that we anticipate today, just a consideration in case future stages also want to use $idLookup in this way.

            Assignee:
            Unassigned
            Reporter:
            Will Buerger
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: