[SERVER-67323] Improve dependency handoff between agg and query Created: 15/Jun/22  Updated: 24/Jan/23

Status: Open
Project: Core Server
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Charlie Swanson Assignee: Backlog - Query Execution
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-66061 Optimize query plans using columnar i... Closed
is related to SERVER-67875 Complete TODO listed in SERVER-66061 Closed
Assigned Teams:
Query Execution
Participants:
Linked BF Score: 35

 Description   

In SERVER-66061 we realized that we are doing some crazy stuff where a $group pipeline may generate a dependency set, pass that as a projection into the query layer, and then have the query layer optimize out that projection if/when a $group is pushed into the query layer. This seems sad and inefficient, so we have the following ideas to explore:
1) Instead of passing dependencies in as a projection, the aggregation system could pass in a new parameter which is the dependencies itself. This would communicate more clearly the intention, and make it possible to only add the projection if we don't swallow the $group.
2) Improve the interface for pushing down the $group so that we can know and decide to push a $project and/or a $group around the same time. Right now the $group is sometimes added in a callback function after we have entered the planner's getExecutorFind(). This makes it sketchy to modify the projection. This could maybe take the form of just always passing a $group into the query system, and letting it decide whether to use SBE to do the $group or to add some shim around DocumentSourceGroup or somehow indicate it was not possible. This may be crazy with library cycles.



 Comments   
Comment by Charlie Swanson [ 22/Nov/22 ]

Throwing this back on the backlog as I transition onto another project. Leaving it in the epic in hopes someone on the team can take a look.

Generated at Thu Feb 08 06:07:50 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.