[SERVER-60800] Allow $search in $lookup/$unionWith Created: 19/Oct/21  Updated: 29/Oct/23  Resolved: 28/Feb/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 6.0.0-rc0

Type: New Feature Priority: Major - P3
Reporter: Charlie Swanson Assignee: Ruslan Abdulkhalikov (Inactive)
Resolution: Fixed Votes: 0
Labels: neweng
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Depends
Documented
is documented by DOCS-15254 Investigate changes in SERVER-60800: ... Closed
Problem/Incident
Related
related to SERVER-71036 Consider resolving variables inside $... Backlog
related to SERVER-62973 Investigate/consider making $$SEARCH_... Closed
related to SERVER-63816 Extend non-correlated pipeline prefix... Closed
related to SERVER-63845 Separate interface to get set of refe... Closed
is related to SERVER-40949 Prevent $lookup into a $sb Subpipeline Closed
is related to SERVER-48120 Optimize a $sort after a $unionWith Backlog
is related to SERVER-65808 $search as subpipeline of $lookup wit... Closed
Backwards Compatibility: Fully Compatible
Backport Requested:
v5.2
Sprint: QO 2021-12-27, QO 2022-01-10, QO 2022-01-24, QO 2022-02-07, QO 2022-02-21, QO 2022-03-07
Participants:
Linked BF Score: 179

 Description   

Allow $search to be used as target of $lookup and $unionWith.

This will provide a better user experience in $search and allow for compatibility with a wider range of use cases.



 Comments   
Comment by Daniele Tassone [ 18/Nov/22 ]

Hi, I am using $search in $lookup.

It might be useful for me to rely on $$correlated_variable defined in let: {}. Unfortunately, it is treated as literal string. Is there any plan to support it in the future? 
Thanks.

Comment by Githook User [ 28/Feb/22 ]

Author:

{'name': 'Ruslan Abdulkhalikov', 'email': 'ruslan.abdulkhalikov@mongodb.com', 'username': 'rusabd1'}

Message: SERVER-60800-enable-lookup-unionwith-nonsharded: Merge remote-tracking branch 'origin/master' into SERVER-60800-enable-lookup-unionwith-nonsharded
Branch: master
https://github.com/mongodb/mongo/commit/ae17599108271d56a8cad70ec9b200929aeec401

Comment by Githook User [ 28/Feb/22 ]

Author:

{'name': 'Ruslan Abdulkhalikov', 'email': 'ruslan.abdulkhalikov@mongodb.com', 'username': 'rusabd1'}

Message: SERVER-60800-enable-lookup-unionwith-nonsharded: Merge remote-tracking branch 'origin/master' into SERVER-60800-enable-lookup-unionwith-nonsharded
Branch: master
https://github.com/10gen/mongo-enterprise-modules/commit/c344f1b95df95a3a9c994e3715b16451ae59a0f7

Comment by Steve Tarzia [ 17/Feb/22 ]

Also note that this ticket's work will not enable $searchMeta in $lookup/$unionWith.  That will be added in https://jira.mongodb.org/browse/PM-2729 

Comment by Githook User [ 08/Feb/22 ]

Author:

{'name': 'ruslan.abdulkhalikov', 'email': 'ruslan.abdulkhalikov@mongodb.com', 'username': 'rusabd1'}

Message: SERVER-60800 fix aggregation_cursor_invalidation tests
Branch: master
https://github.com/mongodb/mongo/commit/aac0bcaa238ecb28dace7d263789608474edb3c6

Comment by Githook User [ 03/Feb/22 ]

Author:

{'name': 'Ruslan Abdulkhalikov', 'email': 'ruslan.abdulkhalikov@mongodb.com', 'username': 'rusabd1'}

Message: SERVER-60800 add optional UUID to ResolvedNamespace
Branch: master
https://github.com/mongodb/mongo/commit/f1664bad02c3ebf8593ae3418fc172e336d8f6cc

Comment by Pawel Terlecki [ 24/Dec/21 ]

i thought about it more. we should actually allow $$SEARCH_META to be defined as a variable associated with the pipeline rooted in $search. if users want to merge $$SEARCH_META from subbranches for unionWith or lookup they should be able to do it explicitly

Comment by Charlie Swanson [ 21/Oct/21 ]

pawel.terlecki it seems like we could do $searchMeta as well, yeah.

Some interesting considerations from a team discussion on 10/21:

  • $lookup could not necessarily be correlated. The arguments to $search are not interpreted as expressions and so something like "$$correlated_variable" would just be a string literal if used within $search.
  • $unionWith would have no guarantees about sort order between the branches. A manual sort could be added but would be in-memory and slow until SERVER-48120 is resolved.
Comment by Pawel Terlecki [ 19/Oct/21 ]

Can we also enable $searchMeta? Say if one wants to union searches in 3 collections and get these 3 counts.

Generated at Thu Feb 08 05:50:46 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.