[SERVER-77427] Avoid going through the network when a shard is targeting only itself for a $lookup subpipeline Created: 24/May/23  Updated: 10/Jan/24  Resolved: 17/Aug/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: 7.1.0-rc0

Type: Improvement Priority: Major - P3
Reporter: Hana Pearlman Assignee: Ivan Fefer
Resolution: Fixed Votes: 0
Labels: pm3229-m1
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Backports
Duplicate
is duplicated by SERVER-75294 Investigate slowness of sharded $lookup Closed
Problem/Incident
causes SERVER-81335 Query operations that avoid going thr... Open
causes SERVER-80194 In sharded_agg_helpers.cpp we retriev... Closed
causes SERVER-80341 sharded_agg_helpers::attachCursorToPi... Closed
Related
related to SERVER-75294 Investigate slowness of sharded $lookup Closed
Assigned Teams:
Query Execution
Backwards Compatibility: Fully Compatible
Backport Requested:
v7.0, v6.0
Sprint: QO 2023-06-12, QO 2023-06-26, QO 2023-07-10, QO 2023-08-07
Participants:
Linked BF Score: 135

 Description   

This is meant to be a targeted improvement specific to $lookup, though depending on the implementation it could affect other commands which use sharded_agg_helpers::attachCursorToPipeline, similar to SERVER-58376. When this work is done, we would expect to see improvement especially for $lookups where the local and foreign data is co-located (see workloads from PERF-4152)



 Comments   
Comment by Githook User [ 26/Sep/23 ]

Author:

{'name': 'Rushan Chen', 'email': 'rushan.chen@mongodb.com', 'username': 'ruchen'}

Message: SERVER-77427 revert
Branch: v7.1
https://github.com/mongodb/mongo/commit/aae297113407b219885bfe44f25a1fb377e04557

Comment by Katya Kamenieva [ 17/Aug/23 ]

🎉

Comment by Githook User [ 16/Aug/23 ]

Author:

{'name': 'Ivan Fefer', 'email': 'ivan.fefer@mongodb.com', 'username': 'Fefer-Ivan'}

Message: SERVER-77427 Use local read if pipeline targets only local shard
Branch: master
https://github.com/mongodb/mongo/commit/5c14aa402e3cbab58d8de1e891f00e91deb023e5

Comment by David Storch [ 31/Jul/23 ]

We should be able to apply this "local reads" optimization for the inner side of $lookup for the following cases:

  • The collection is unsharded and untracked by the sharding catalog, but the $lookup is executing on the primary shard.
  • The collection is unsharded but tracked by the sharding catalog and is placed locally on the node where $lookup is executing.
  • The collection is sharded but all chunks are local.
  • The collection is sharded with chunks distributed across shards. But after targeting based on the query predicate, we find that the shard executing the $lookup only needs to target itself.
Generated at Thu Feb 08 06:35:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.