[SERVER-17943] add $filter expression to $project to work with $map and $reduce Created: 08/Apr/15  Updated: 17/Jan/18  Resolved: 07/May/15

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 3.1.3

Type: New Feature Priority: Major - P3
Reporter: Asya Kamsky Assignee: Charlie Swanson
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by CSHARP-1315 Refactor LINQ infrastructure to enabl... Closed
Duplicate
is duplicated by SERVER-6612 Support projecting multiple array val... Closed
is duplicated by SERVER-9370 Optimize $match $unwind $match sequen... Closed
is duplicated by SERVER-14876 support $elemMatch in aggregation $pr... Closed
Related
related to SERVER-18423 Typo in aggregation $map and $filter ... Closed
related to CSHARP-1405 add $filter expression to $project to... Closed
related to DRIVERS-234 Aggregation Builder Support for 3.2 Closed
Backwards Compatibility: Fully Compatible
Sprint: Quint Iteration 3
Participants:

 Description   

Similar to $map (and in combination with possible $reduce SERVER-17258) adding $filter would complete the generic framework for handling any sort of array processing during projection in agg.



 Comments   
Comment by Asya Kamsky [ 25/Aug/15 ]

oleg@evergage.com

My apologies, I was too terse in my description. When I said "new features" I meant that a general rule about whether to backport any feature is whether it's a new user facing feature that requires the drivers (among other things) to provide support for it. While I understand that new aggregation operator can be used without special support from the drivers, nonetheless our new feature checklist includes noting whether drivers need to add any specific support for the feature (as an example, see CSHARP-1315 which is refactoring LINQ code to be able to add support for new aggregation operators, stages, etc).

I understand your frustration when you see that a relatively isolated new functionality is added to a component in the upcoming version that you may badly need in the current version - but consider the amount of work required if part of adding this new feature included refactoring or improving parts of our test framework and there isn't a way to backport the feature with its tests without making major changes to current code. Worse yet may be if the new feature turns out to have some very bad performance side-effect or even cause a crash - it's much simpler to manage that just in the version that the feature is released in, especially since this new feature won't be released to users till 3.2 is GA.

I don't believe that we are unique in this approach - patch or dot releases are meant for only minor changes, usually reserved for bug fixes, performance optimizations, security patches and such.

Asya Kamsky

Comment by Oleg Rekutin [ 24/Aug/15 ]

asya, I apologize if this is not an appropriate place to register this comment, but could you please ask MongoDB leadership to reflect on this "no new features" policy.

I've noticed many a regression in WiredTiger performance or functionality (just look at all the 3.0.x releases, specifically this new 3.0.6 release and the things that it fixes). E.g. the side effect of fixing WT-1989 producing a critical bug (SERVER-19673) that should have been a stop-ship on the 3.0.5 series, and an immediate 3.0.6 with only that fix should've been released. With all these WT regressions, I find it hard to believe that the WT branch for MongoDB 3.0 is obeying the "no new features" rule internally.

$filter and $arrayElemAt (SERVER-4589) are critical tools for being able to extract sufficiently fast aggregation performance out of MongoDB for when objects contain array data. We've been running with an a custom $elemMatch implementation on 2.4, and now with a backport of these features ($filter and $arrayElemAt in $project) on 2.6 and 3.0.

These are lightweight, low-complexity, low-risk backports that enable aggregations that are simply impossible in anything other than 3.1. Alternatives in 3.0 and below are MapReduce (too slow) and "$unwind/$group by $id" stages, which is an expensive group that has performance limits.

My main point is that since the development process is not actually producing a 3.0.x branch that's sufficiently stable release-to-release, then it's a shame/unfortunate to watch small, but very useful features get left behind into a future 3.2.x release (including all the waiting time for the 3.2.x branch to be truly production-ready with a few point releases under its belt).

Anyway, if anyone is interested in building your own backports:

3.0 backport: https://github.com/evergage/mongo/commit/be4e427de6df5ebe81016fe87f3a12af80782437
2.6 backport: https://github.com/evergage/mongo/commit/c6b8dba7f134de5cd698260efb56ad5c2ccc2a39 and https://github.com/evergage/mongo/commit/31fca056b3be1b4fec7c003f3ddf0a61e42e498d

Comment by Asya Kamsky [ 20/Aug/15 ]

oleg@evergage.com

Unfortunately not - we can't back port completely new features.

Comment by Oleg Rekutin [ 20/May/15 ]

Is this something that would be considered as a backport to 3.0 or 2.6 series?

Comment by Githook User [ 07/May/15 ]

Author:

{u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'cswanson310@gmail.com'}

Message: SERVER-17943 Add $filter aggregation expression
Branch: master
https://github.com/mongodb/mongo/commit/6b38c7a53f2e284583199c12b4b9f6cd8d69004a

Generated at Thu Feb 08 03:46:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.