[SERVER-53638] Enable pushdown of config.cache.chunks $lookup through $sort Created: 07/Jan/21 Updated: 29/Oct/23 Resolved: 12/Apr/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | 5.0.0-rc0 |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Charlie Swanson | Assignee: | Yuhong Zhang |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | query-work-resharding | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||||||||||||||
| Issue Links: |
|
||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||
| Sprint: | Query Optimization 2021-02-22, Query Optimization 2021-03-08, Query Optimization 2021-03-22, Query Optimization 2021-04-05, Query Optimization 2021-04-19 | ||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||
| Description |
|
As part of a resharding operation we issue a pipeline with a $sort followed by a $lookup on config.cache.chunks. For efficiency and resumability, we need each shard to provide things in a sorted order and also perform the $lookup. Today this doesn't work because the $sort forces a split of a sharded pipeline and does not push the $lookup into the shards part. For this particular $lookup, we know it is safe to run it in parallel, so we should be able to push it into the shards pipeline. |
| Comments |
| Comment by Githook User [ 09/Apr/21 ] |
|
Author: {'name': 'Charlie Swanson', 'email': 'charlie.swanson@mongodb.com', 'username': 'cswanson310'}Message: Co-authored-by: Yuhong Zhang <danielzhangyh@gmail.com> |
| Comment by Charlie Swanson [ 18/Mar/21 ] |
|
Uploaded my WIP: optimize_sort_lookup.patch |
| Comment by Charlie Swanson [ 07/Jan/21 ] |
|
Anyone who picks this up: be aware of some similar discussion of tradeoffs in SERVER-26442. As always, we should try to avoid making things worse with this optimization. |