[SERVER-18966] Allow exclusion in $project stage of aggregation pipeline Created: 13/Jun/15 Updated: 06/Apr/23 Resolved: 24/Jun/16 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | 3.3.9 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Asya Kamsky | Assignee: | Charlie Swanson |
| Resolution: | Done | Votes: | 3 |
| Labels: | expression, stage, usability | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Backwards Compatibility: | Minor Change | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Sprint: | Query 11 (03/14/16), Query 12 (04/04/16), Query 13 (04/22/16), Query 14 (05/13/16), Query 15 (06/03/16), Query 16 (06/24/16) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Participants: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Description |
| Comments |
| Comment by tony kerz [ 11/Jan/17 ] | ||||||||||
|
doh, yeah, invalid json, and SERVER-27614 would be great for this. thanks! | ||||||||||
| Comment by Charlie Swanson [ 09/Jan/17 ] | ||||||||||
|
Hi tony-kerz, I understand. Yes, it would be useful to do so. That syntax looks reasonable, but isn't valid JSON (there's no field name for the $ifNotNull object). I think the syntax proposed in | ||||||||||
| Comment by tony kerz [ 09/Jan/17 ] | ||||||||||
|
thanks for the response charlie. your comments around the risk of your workaround make sense. i was just trying to keep unnecessary bloat out of the resultant records, take the example where a field is non-null in only some small percentage of the records, it seems like a waste to populate all records with:
how would you feel about something like:
to handle that case? | ||||||||||
| Comment by Charlie Swanson [ 09/Jan/17 ] | ||||||||||
|
Hi tony-kerz, I don't know of any great way to do that. I'm curious why you want to do this though? Are you using 'b' later in the pipeline, or do you just want to avoid serializing it if you aren't going to be using it? One weird workaround would be to do something like
which would not include 'b' if it was null or missing. Note that this would have some ramifications earlier in the pipeline, since the dependency analysis would now think that a field called 'nonExistentField' was necessary to answer this pipeline. Essentially, this could mean that if your pipeline was previously using a covered index scan on 'a' and 'b', that index could no longer provide a covered projection, since we would need to fetch the documents to find the field 'nonExistentField'. So this is not ideal, but if your $project is after a $group or something like that, it wouldn't matter as much. | ||||||||||
| Comment by tony kerz [ 07/Jan/17 ] | ||||||||||
|
thanks charlie, that makes sense. what i was really after is a little bit of a spin on that theme where i wanted to exclude a key/value if the value was null,
which would only project b if it has a non-null value... do you know of any way to accomplish something like that? | ||||||||||
| Comment by Charlie Swanson [ 06/Jan/17 ] | ||||||||||
|
Hi tony-kerz, Your projection is mixing an inclusion (the {a: 1} part) and an exclusion (the {b: false} part). Mixing inclusion and exclusion is illegal, since it is ambiguous whether or not fields not named (such as "_id" in your case, or a field "c" if your documents had one) should be included or not. If you only want the field "a", just stick to {a: 1}. If you want everything except "b", just use {b: 0} or {b: false}. Hope this helps. | ||||||||||
| Comment by tony kerz [ 06/Jan/17 ] | ||||||||||
|
i'm currently running 3.4.1 and seeing this:
what am i missing? | ||||||||||
| Comment by Charlie Swanson [ 24/Jun/16 ] | ||||||||||
|
The following changes might be considered "backwards breaking":
Additional changes in behavior:
| ||||||||||
| Comment by Githook User [ 24/Jun/16 ] | ||||||||||
|
Author: {u'username': u'cswanson310', u'name': u'Charlie Swanson', u'email': u'charlie.swanson@mongodb.com'}Message: | ||||||||||
| Comment by Dissatisfied Former User [ 13/Jun/16 ] | ||||||||||
|
While I agree that this would be a workaround for SERVER-22787, I don't think this ticket duplicates the other one. This ticket has far more broad implications than adding a special flag to the $lookup stage. | ||||||||||
| Comment by Jose Antonio Illescas Olmo [ 23/Apr/16 ] | ||||||||||
|
This feature (remove fields on $projection stage) can be a workaround for SERVER-22787 ($lookup.excludeLocalField) | ||||||||||
| Comment by Jose Antonio Illescas Olmo [ 12/Apr/16 ] | ||||||||||
|
I need this feature, All my documents contains "lastModified" date exposed as Last-Modified http-header (coditional get and put) => this field must be removed from document/json content Thank you |