[SERVER-39938] aggregation $match before $lookup optimization doesn't happen when $expr: $eq is used Created: 04/Mar/19 Updated: 29/Oct/23 Resolved: 31/Mar/21 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | 4.0.6 |
| Fix Version/s: | 5.0.0-rc0 |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Asya Kamsky | Assignee: | Alya Berciu |
| Resolution: | Fixed | Votes: | 1 |
| Labels: | optimization, query-44-grooming | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||
| Operating System: | ALL | ||||||||||||||||||||
| Sprint: | Query 2019-12-30, Query Optimization 2021-02-22, Query Optimization 2021-03-08, Query Optimization 2021-03-22, Query Optimization 2021-04-05 | ||||||||||||||||||||
| Participants: | |||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||
| Linked BF Score: | 120 | ||||||||||||||||||||
| Description |
|
We check if $match has an expression that isn't on "as" and move it before $lookup - if it has an expression on "as.x" we move it into $lookup. Expected behavior:
Using $expr for x:5 breaks the $lookup matching optimization though equality for x:5 still gets moved before $lookup but does not get pushed into cursor
|
| Comments |
| Comment by Alya Berciu [ 31/Mar/21 ] |
|
Closing as the fix has been merged. |
| Comment by Githook User [ 31/Mar/21 ] |
|
Author: {'name': 'Alya Berciu', 'email': 'alyacarina@gmail.com', 'username': 'alyacb'}Message: |
| Comment by David Storch [ 17/Dec/19 ] |
|
If I understand correctly, the patch I'm working on for SERVER-44258 would address this. I'll try to confirm once I have a satisfactory fix for SERVER-44258. |
| Comment by James Wahlin [ 05/Mar/19 ] |
|
Currently when aggregation pipeline optimization takes place, we first optimize for relationships in the pipeline (splitting, merging and moving stages) and then perform stage-specific optimization. This removes any $match with $expr/$eq from consideration for split/merge/move as we will not consider its pre-optimized form (meaning prior to rewrite of the $expr:$eq to $and: [$expr:$eq, $_internalEqMatchExpression]). Additionally for the example of above, we would only consider pushing the {"a.z":10} match into $lookup, if the $expr has been split and moved as we only considering pushdown for the entire MatchExpression and will not split for this case alone. Both of the above could be fixed if we allowed a $expr MatchExpression to be split/merged/moved prior to optimization. This solution would also optimize $expr statements with comparisons other than $eq, pushing the comparison earlier. An alternative fix would be to optimize $match with $expr prior to the pipeline-level optimization, allowing the split & move to take place. This would still leave us with the $lookup pushdown issue, as only $internEqMatchExpression would be moved leaving the $expr paired with the "a.z":10 statement. |
| Comment by James Wahlin [ 04/Mar/19 ] |
|
Currently the only aggregation expression that we will rewrite to the match language is $eq. Without such a rewrite, expressions embedded in a $expr will not be eligible for use in our optimizations. This includes both repositioning match statements in the pipeline and push to the query engine. While in most cases we would not be able push range based operators such as $gt and $lt to the query engine (due to differences in type bracketing between match and aggregation) it would be fine to reposition in the pipeline when valid. asya - I created |