[SERVER-28705] Add optimization to execute $lookup on local shards when possible Created: 10/Apr/17  Updated: 06/Dec/22  Resolved: 30/Sep/21

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Sharding
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Mohannad Saeed Assignee: Backlog - Query Optimization
Resolution: Duplicate Votes: 2
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-29159 Allow "from" collection of $lookup to... Closed
Gantt Dependency
has to be done after SERVER-29159 Allow "from" collection of $lookup to... Closed
Related
related to SERVER-27496 allow self-$lookup on shard key value... Backlog
is related to SERVER-22384 Add $lookUpOne to retrieve single Obj... Open
Assigned Teams:
Query Optimization
Participants:

 Description   

Performing joins using local shard indices is cheap. In this setup, parent-child relationships can be implemented as long parent and child documents live on the same shard through proper routing by application developers. This feature is similar to ElasticSearch has_parent and has_child queries. $joinOne works with a single object while $joinMany works with an array of objects.



 Comments   
Comment by Charlie Swanson [ 12/May/17 ]

Hi all,

I'd like to clarify a couple things. For one, the $joinOne part of this ticket has already been requested in SERVER-22384, so I will remove that part of this request so that it can be tracked independently. Please add comments related to that request on SERVER-22384 to help us track the discussion. Second, I don't think I understand this request given that $lookup cannot look up into sharded collections:

The from collection cannot be sharded.

Knowing this, I can't think of any case where the described optimization makes sense, so I'd like to close this as "Won't Fix", or maybe mark it as 'depends on' SERVER-29159. Further, even if the 'from' collection were allowed to be sharded, there is no way that I know of to define a relationship between the shard keys of two collections to guarantee that some sort of parent/child or primary key/foreign key relationship will always be co-located.

Comment by Mohannad Saeed [ 11/Apr/17 ]

Hi, I think in MongoDB terminology, if both the local and foreign fields of the $lookup operation on all documents to be joined share the same shard key then this means that this operation can happen during scatter phase locally on shards, correct? If yes then the linked issue is very similar to what I am asking.

Comment by Kelsey Schubert [ 10/Apr/17 ]

Hi geekox86,

Thank you for the feature request. I've linked a related ticket, SERVER-27496, would you please review it and see if it meets your requirements. If not, would you please clarify how these commands would behave, and give an example of their usage?

Kind regards,
Thomas

Generated at Thu Feb 08 04:18:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.