[SERVER-32308] Add the ability for a $lookup stage to execute on mongos against a sharded foreign collection Created: 13/Dec/17  Updated: 30/Oct/23  Resolved: 14/Dec/18

Status: Closed
Project: Core Server
Component/s: Aggregation Framework, Querying
Affects Version/s: None
Fix Version/s: 4.1.7

Type: New Feature Priority: Major - P3
Reporter: David Storch Assignee: James Wahlin
Resolution: Fixed Votes: 4
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by SERVER-38728 Allow pipeline with $lookup into a sh... Closed
is depended on by SERVER-32309 Serialize variables within nested pip... Closed
is depended on by SERVER-32666 Allow $graphLookup against a sharded ... Closed
is depended on by SERVER-39238 Allow foreign $lookup pipeline split ... Backlog
Documented
is documented by DOCS-12271 Docs for SERVER-32308: Add the abilit... Closed
Backwards Compatibility: Fully Compatible
Sprint: Query 2018-01-01, Query 2018-01-15, Query 2018-12-17
Participants:
Linked BF Score: 100

 Description   

In order to support $changeStreams "updateLookup" functionality (see the documentation here), the aggregation subsystem now has machinery for $lookup-like query execution on mongos.

We should extend this functionality to work for the $lookup stage itself:

  • When the foreign collection is sharded, split the pipeline such that the $lookup is part of the merging pipeline. Execute the merging pipeline on mongos, performing lookups across the network to the shards as needed.
  • If the foreign collection is not sharded, send the merging part of the pipeline to the primary shard and perform local lookups, just as we do in all existing versions of MongoDB.
  • If the merging part of the pipeline containing the $lookup cannot execute on mongos, raise an error to the application.
  • Support only localField/foreignField $lookup for now; adding support for $lookup with a sub-pipeline into a sharded foreign collection is future work.


 Comments   
Comment by Githook User [ 14/Dec/18 ]

Author:

{'username': 'jameswahlin', 'email': 'james@mongodb.com', 'name': 'James Wahlin'}

Message: SERVER-32308 Address ASAN failure due to leaked Pipeline in attachCursorSourceToPipeline
Branch: master
https://github.com/mongodb/mongo/commit/7416ce3c3e6e1a84d8ccc111bf1cd3895bada6f3

Comment by James Wahlin [ 14/Dec/18 ]

Reopening to address memory leak when an exception is thrown in attachCursorSourceToPipeline().

Comment by Githook User [ 13/Dec/18 ]

Author:

{'name': 'Benety Goh', 'email': 'benety@mongodb.com', 'username': 'benety'}

Message: SERVER-32308 remove unused variable
Branch: master
https://github.com/mongodb/mongo/commit/6b0e401683ad1bc3938a1111a95750fce96e2d72

Comment by James Wahlin [ 12/Dec/18 ]

Reopening to address BF-11527, a compile_all failure on the "* Enterprise Windows 2008R2 compile_all" and "* Windows 2008R2 DEBUG" due to an unused variable in document_source_lookup_change_post_image_test.cpp.

Comment by Githook User [ 12/Dec/18 ]

Author:

{'name': 'James Wahlin', 'email': 'james@mongodb.com', 'username': 'jameswahlin'}

Message: SERVER-32308 Support for $lookup to execute on mongos against a sharded foreign collection
Branch: master
https://github.com/mongodb/mongo/commit/056d61676f91f6da0a030347ae4b92255d752d8f

Comment by Nicholas Zolnierz [ 16/Jan/18 ]

Re-opening this ticket as we're currently re-evaluating the priority of the sharded $lookup project.

Comment by Githook User [ 16/Jan/18 ]

Author:

{'email': 'nicholas.zolnierz@mongodb.com', 'name': 'Nick Zolnierz', 'username': 'nzolnierzmdb'}

Message: Revert "SERVER-32308: Add the ability for a $lookup stage to execute on mongos against a sharded foreign collection"

This reverts commit 7298d273c0497f2720ec1471ad0f4910bff07af4.
Branch: master
https://github.com/mongodb/mongo/commit/4eabf1ea6225f444b3b0b3b2fee785aaa306212f

Comment by Githook User [ 12/Jan/18 ]

Author:

{'email': 'nicholas.zolnierz@mongodb.com', 'name': 'Nick Zolnierz', 'username': 'nzolnierzmdb'}

Message: SERVER-32308: Add the ability for a $lookup stage to execute on mongos against a sharded foreign collection
Branch: master
https://github.com/mongodb/mongo/commit/7298d273c0497f2720ec1471ad0f4910bff07af4

Generated at Thu Feb 08 04:29:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.