[SERVER-31747] Allow projections inside lookup stages in aggregation pipelines Created: 27/Oct/17  Updated: 06/Dec/22  Resolved: 27/Oct/17

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Jose Assignee: Backlog - Query Team (Inactive)
Resolution: Duplicate Votes: 0
Labels: feature
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-29072 Add support for $lookup into a sub-pi... Closed
Assigned Teams:
Query
Backwards Compatibility: Major Change
Participants:

 Description   

Now when you have a lookup stage in your aggregation pipeline, that lookup retrieves all the complete documents from the foreign collection, this is risky because you can reach the maximum document size very easily if:

  1. The foreign collection is big enough
  2. Each document is very heavy (our case)

It may be legit to increase the maximum document size limit when you need all the data from the foreign collection documents, but you will finally reach the limit at some point, and, for the second point, not always all the data holding from a document is needed, just a small portion of it, but, if you cannot make a projection from each document, discarding what you don't need, then the maximum document size will be reached as well at some point. In fact the limit could be always reached, but it is harder to reach that limit when you have a bunch of numbers, than having a bunch of other huge data.

We think that the solution is to allow projections inside the lookup stage.



 Comments   
Comment by James Wahlin [ 27/Oct/17 ]

Hi Jose,

The feature you request will be available in MongoDB 3.6.0 and can be tested in our 3.6.0-rc1 release candidate. This comes as part of a new $lookup feature which allows users to specify a foreign pipeline. For details see the following docs page:

https://docs.mongodb.com/master/reference/operator/aggregation/lookup/#join-conditions-and-uncorrelated-sub-queries

Thanks,
James

Comment by Mark Agarunov [ 27/Oct/17 ]

Hello jgl,

Thank you for the detailed description. I've set the fixVersion to "Needs Triage" for this new feature to be scheduled against our currently planned work. Updates will be posted on this ticket as they happen.

Thanks,
Mark

Generated at Thu Feb 08 04:28:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.