[SERVER-38560] $graphLookup doesn't support search within specific field(like SQL based grammar of 'select') Created: 12/Dec/18  Updated: 22/May/23

Status: Backlog
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Yin Ting Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 8
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by SERVER-40205 Allow $graphLookup to use projections... Closed
Related
Assigned Teams:
Query Optimization
Participants:
Case:

 Description   

As designed, $graphLookup is only capable of traversing a graph recursively, that is, by performing repeated queries on the "connectToField" value. However, if documents are huge because of many other fields that are not important(want to be ignored) for current graphlookup search, this will lead to the search exceeding the maximum memory usage. Leverage disk during the search will make bad performance.   $graphLookup should support an option that accepts a select within fields(very similar to  $project.), and this could be applied at each stage of the breadth-first search.  

For example, consider a set of flight data, where each document represents a flight between two cities:

 

{ from: '1', to: '2' , oi: 'hugeJsonString1' }
{ from: '2', to: '3', oi: 'hugeJsonString1' }
{ from: '4', to: '5',  oi: 'hugeJsonString1' }
...

 

If a user wishes to figure out all cities they can fly to from New York within two stops, but he doesn't care about the 'oi' part, they could add a 'project' keyword to exclude it during each stage of graphlookup search, like below: 

 

{$graphLookup: {| |from: 'flights',| |project: \{'oi': 0}

,

connectToField: 'from',
connectFromFIeld: 'to',
as: 'destinations'
}}


 Comments   
Comment by Dennis Hoefakker [ 30/Jun/20 ]

This is really needed so a good implementation of $graphlookup can be done. In "real" collections there is a good possibility for a out of memory. When there is a option to "$project" the fields of the related documents this can be eliminated. I'm now forced to implement a custom solution because there is no such feature in place. I was really expecting this to be in place since we are almost on version 4.4 now (where $graphlookup was implemented in 3.4).

Comment by Kelsey Schubert [ 14/Dec/18 ]

Thank you for the feature request, tyin. I've marked this ticket for consideration by the Query Team.

Kind regards,
Kelsey

Generated at Thu Feb 08 04:49:18 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.