[SERVER-53638] Enable pushdown of config.cache.chunks $lookup through $sort Created: 07/Jan/21  Updated: 29/Oct/23  Resolved: 12/Apr/21

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 5.0.0-rc0

Type: New Feature Priority: Major - P3
Reporter: Charlie Swanson Assignee: Yuhong Zhang
Resolution: Fixed Votes: 0
Labels: query-work-resharding
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Text File optimize_sort_lookup.patch    
Issue Links:
Depends
is depended on by SERVER-53940 Support resuming resharding collectio... Backlog
Problem/Incident
causes SERVER-57164 Invariant failure in a $group-by-vari... Closed
Related
related to SERVER-56583 Push $setWindowFields to shards when ... Backlog
is related to SERVER-26442 Push $sort before $project and $addFi... Open
Backwards Compatibility: Fully Compatible
Sprint: Query Optimization 2021-02-22, Query Optimization 2021-03-08, Query Optimization 2021-03-22, Query Optimization 2021-04-05, Query Optimization 2021-04-19
Participants:

 Description   

As part of a resharding operation we issue a pipeline with a $sort followed by a $lookup on config.cache.chunks. For efficiency and resumability, we need each shard to provide things in a sorted order and also perform the $lookup. Today this doesn't work because the $sort forces a split of a sharded pipeline and does not push the $lookup into the shards part. For this particular $lookup, we know it is safe to run it in parallel, so we should be able to push it into the shards pipeline.



 Comments   
Comment by Githook User [ 09/Apr/21 ]

Author:

{'name': 'Charlie Swanson', 'email': 'charlie.swanson@mongodb.com', 'username': 'cswanson310'}

Message: SERVER-53638 Enable pushdown of config.cache.chunks $lookup through $sort

Co-authored-by: Yuhong Zhang <danielzhangyh@gmail.com>
Branch: master
https://github.com/mongodb/mongo/commit/0e967d6ed33cdf9eb8314e6ae3fb3e2261e213d7

Comment by Charlie Swanson [ 18/Mar/21 ]

Uploaded my WIP: optimize_sort_lookup.patch. I'm going to hand this over to yuhong.zhang to put the finishing touches on, since I haven't found a lot of time to work on this recently. 

Comment by Charlie Swanson [ 07/Jan/21 ]

Anyone who picks this up: be aware of some similar discussion of tradeoffs in SERVER-26442. As always, we should try to avoid making things worse with this optimization.

Generated at Thu Feb 08 05:31:28 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.