[SERVER-17258] Add $reduce expression operator for rollup of arrays Created: 11/Feb/15 Updated: 13/Dec/17 Resolved: 21/Apr/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | 3.3.5 |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Jon Rangel (Inactive) | Assignee: | Benjamin Murphy |
| Resolution: | Done | Votes: | 9 |
| Labels: | expression, neweng | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | Query 13 (04/22/16) | ||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||||||||||||||||||||||||||
| Description |
|
Syntax
Examples
Notes
Errors
Old Description Along similar lines, it would also be useful to have a $reduce operator to rollup an array into a scalar element using a specified combining function. For example, to return the sum of all array elements. To illustrate: Given an input of:
We should return:
This is currently possible using a relatively expensive $unwind and $group (plus some additionally voodoo to handle unwinding of the empty arrays), but it would be more elegant, and likely more performant, to do something like the following:
[Note: the above proposed syntax is just for illustrative purposes] |
| Comments |
| Comment by Benjamin Murphy [ 21/Apr/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This ticket introduces the $reduce operator, which should be documented as an aggregation expression. In addition, any drivers that provide aggregation helpers should be updated to include support for $reduce. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Githook User [ 21/Apr/16 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Author: {u'username': u'benjaminmurphy', u'name': u'Benjamin Murphy', u'email': u'benjamin_murphy@me.com'}Message: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Asya Kamsky [ 29/Apr/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
antoine.hom@amadeus.com it looks likely that this feature (at least some of the variants of it, sum, min, max, etc) will be scheduled for the next major release, so I don't think it's worthwhile for you to spend any effort on it. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Antoine Hom [ 14/Apr/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hey, I was wondering if that would help get this going if I made a pull request related to this ticket. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Antoine Hom [ 23/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
After looking at our tests it was indeed a slightly different scenario than the one here. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Asya Kamsky [ 21/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
You should be able to run [ { .. pipeline .. }], {explain:true}to see what fields would be passed through - with project first, the pipeline will go with what you explicitly specify, without it it will pass through minimum necessary fields. It would be unexpected for the two to be different duration executions, but would be good to know if they are. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Antoine Hom [ 21/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hey Asya, A simple $sum is also completely fine, note that I also have examples where $max/$min would be very helpful and avoid unnecessary $unwind/$group, I would gladly post them in their short versions if you want to see them. For the other $group operators, we can either work around them ($avg => $sum/$size) or they would be better answered by other feature requests [$first/$last => $slice SERVER-6074] [$push => $concat SERVER-14872] IIRC, we are always adding a project as soon as possible in our pipelines because we detected some difference in performances. I will take a look at it on monday when I have access to the tests we did in the past but our setup is quite unusual so that may be related to that. Cheers, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Asya Kamsky [ 20/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Please note that your aggregation pipeline is already longer than it needs to be as you have $project stages that don't actually compute any new fields. I recommend that you remove those as they are unnecessary (aggregation will only pass through the fields it knows will be needed) and may in some cases slow things down. With proposed $sum your pipeline would become:
or if you prefer terser syntax even:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Asya Kamsky [ 20/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Would it be sufficient to have a projection operator $sum to $add an array of values? We would probably have an analogous $product for $multiply. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Antoine Hom [ 20/Mar/15 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Here is one concrete example with how we can answer it right now and how it would be answered with such a $reduce operator: Sample query:
Sample document:
Aggregation without reduce: Issues are:
Aggregation with reduce:
|