[SERVER-25957] Optimize $filter + $arrayElemAt to avoid scanning entire array Created: 04/Sep/16 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | 2.6.4 |
| Fix Version/s: | None |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Xavier Del Castillo | Assignee: | Backlog - Query Optimization |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | neweng, optimization, qopt-team | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Assigned Teams: |
Query Optimization
|
||||||||
| Participants: | |||||||||
| Description |
| Comments |
| Comment by Charlie Swanson [ 13/Dec/19 ] | |||||
|
The above commit message mentioned the wrong SERVER ticket so this was mistakenly closed. There is a work in progress pull request here. | |||||
| Comment by Craig Homa [ 13/Dec/19 ] | |||||
|
Re-opening this as it was closed in error. | |||||
| Comment by Githook User [ 01/May/18 ] | |||||
|
Author: {'email': 'KevinCybura@gmail.com', 'name': 'KevinCybura'}Message: SERVER-25957 Optimize $indexOfArray when array argument is constant. Signed-off-by: Charlie Swanson <charlie.swanson@mongodb.com> Closes #1229 | |||||
| Comment by Asya Kamsky [ 29/Mar/18 ] | |||||
|
Please note that:
This should not be done with $filter in aggregation expressions, but with $in. | |||||
| Comment by Asya Kamsky [ 28/Jun/17 ] | |||||
|
This may be useful as a general optimization not just for $arrayElemAt but for any sort of $slice of the result array. In other words, it would be useful to be able to short circuit out of $filter early if the full result array will not be used.
The above should be able to stop processing "$bigArray" in $filter as soon as 5 elements have been generated from $filter expression. | |||||
| Comment by Xavier Del Castillo [ 28/Sep/16 ] | |||||
|
Thanks for considering this request. | |||||
| Comment by Charlie Swanson [ 09/Sep/16 ] | |||||
|
xdc, This seems like a legitimate request, at least as a performance improvement. As you noticed above, you can use $filter to do most of the work, so if you combine that with an {$arrayElemAt: [{$filter: ...}, 0]} it should do what you want. I'll convert this ticket into a request for the above expression to be optimized not to scan the entire array, instead stopping at the first matching element. | |||||
| Comment by Xavier Del Castillo [ 04/Sep/16 ] | |||||
|
This is a clone task of: The resolution was that: $filter was implemented on: https://jira.mongodb.org/browse/SERVER-17943 However, I do not think `$filter` and `$elemMatch` are equal - according to the docs `$elemMatch` stops looking when it matches at least one, potentially stopping scanning early - however `$filter` will scan the whole array, even if the first element in the array matches the condition, returning the subset of elements passing the condition. For answering the question "Is there an element in this array that matches this?" `$elemMatch` is the correct choice, `$filter` can answer that question but does unnecessary work. Thoughts on the above use case? |