[SERVER-5781] Implement $addFields aggregation stage for using expression language to add new fields to a document Created: 07/May/12  Updated: 22/Mar/17  Resolved: 12/Aug/16

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: 3.3.11

Type: New Feature Priority: Major - P3
Reporter: Patrick Kaeding Assignee: Carly Robison
Resolution: Done Votes: 26
Labels: stage, usability
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
is depended on by CSHARP-1750 Add support for $addFields aggregatio... Closed
is depended on by JAVA-2283 Add builder for $addFields aggregatio... Closed
Documented
is documented by DOCS-8721 Document 3.4 $addFields stage Closed
Duplicate
is duplicated by SERVER-24779 Add a $transform aggregation stage Closed
Related
related to SERVER-25200 $project computed fields on arrays se... Closed
related to SERVER-25581 Leaked ParsedAddField when destructin... Closed
related to DRIVERS-297 Aggregation Framework Support for 3.4 Closed
related to SERVER-8582 Extend document expression language i... Closed
is related to SERVER-24743 $group aggregation command should be ... Closed
is related to SERVER-18966 Allow exclusion in $project stage of ... Closed
Backwards Compatibility: Fully Compatible
Sprint: Integration 2016-08-29
Participants:
Linked BF Score: 0

 Description   

As of 3.3.11, we added the $addFields aggregation stage:

Syntax

{$addFields: {newFieldName1: <Expression1>, ...}}

Examples

// =====  Example #1 - Adding a literal field value =====
>db.example.insert({_id: 0, a: 1})
>db.example.aggregate([{$addFields: {newField: “hello”}}]);
{_id: 0, a: 1, newField: “hello”}
 
// =====  Example #2 - Adding a value from a field path =====
>db.example.insert({_id: 0, a: 1})
>db.example.aggregate([{$addFields: {b: “$a”}}])
{_id: 0, a: 1, b: 1}
 
// =====  Example #3 - Adding a value from a system variable =====
>db.example.insert({_id: 0, a: 1})
>db.example.aggregate([{$addFields: {this: “$$CURRENT”}}])
{_id: 0, a: 1, this: {_id: 0, a: 1}}
 
// =====  Example #4 - Adding a value from an expression object =====
>db.example.insert({_id: 0, a: 1})
>db.example.aggregate([{$addFields: {myNewField: {c: 3, d: 4}}}])
{_id: 0, a: 1, myNewField: {c: 3, d: 4}}
 
// =====  Example #5 - Adding a value from an operator expression =====
>db.example.insert({_id: 0, a: 1})
>db.example.aggregate([{$addFields: {alt3: {$lt: [“$a”, 3]}}}])
{_id: 0, a: 1, alt3: true}
 
// =====  Example #6 Adding multiple new fields at once =====
>db.example.insert({_id: 0, a: 1})
>db.example.aggregate([{$addFields: {b: 3, c: 5}}])
{_id: 0, a: 1, b: 3, c: 5}
 
// =====  Example #6 - Setting a field that already exists =====
>db.example.insert({_id: 0, a: 1})
>db.example.aggregate([{$addFields: {a: [1, 2, 3]}}])
{_id: 0, a: [1, 2, 3]}

Original Description

It would be great if you could use $project (or something) to decorate the input object with additional, calculated values. This way, you wouldn't need to explicitly name all the fields that you want to carry forward if you just need to add calculated values.

For example, currently, in a collection with the following document:

{_id: 0, foo: 1, bar: 1}

When you run the following pipeline:

db.coll.aggregate([{$project: {newField: {$add: [1, '$foo']}}}])

You get the following document:

{_id: 0, newField: 2}

This is requesting that there be some way to get the following document (with all original fields present), without knowing which fields are in the original document:

{_id: 0, foo: 1, bar: 1, newField: 2}

This could be accomplished by adding some option to the $project stage, e.g.

db.coll.aggregate([{$project: {newField: {$add: [1, '$foo']}, $keepExisting: true}}])

Or by adding a new stage, e.g.

db.coll.aggregate([{$set: {newField: {$add: [1, '$foo']}}])



 Comments   
Comment by Max Hirschhorn [ 12/Aug/16 ]

Re-resolving this ticket. The memory leak will be fixed under SERVER-25581 after 3.3.11 is released.

Comment by Max Hirschhorn [ 12/Aug/16 ]

Reopening to address a memory leak when deleting the DocumentSource created for the $addFields stage.

Comment by Githook User [ 11/Aug/16 ]

Author:

{u'username': u'carlyrobison', u'name': u'Carly Robison', u'email': u'crobison@caltech.edu'}

Message: SERVER-5781 fixed unittest error code for addFields
Branch: master
https://github.com/mongodb/mongo/commit/8238984db6727c13797b88b5a53b1cd210e0fb11

Comment by Githook User [ 11/Aug/16 ]

Author:

{u'username': u'carlyrobison', u'name': u'Carly Robison', u'email': u'crobison@caltech.edu'}

Message: Revert "SERVER-5781 fixed unittest error code for addFields"

This reverts commit 4fc451a03f06b696ec563144385aabe2d3d619cf.
Branch: master
https://github.com/mongodb/mongo/commit/54c295e43229197db8741ad30ecef33aaca59798

Comment by Githook User [ 11/Aug/16 ]

Author:

{u'username': u'carlyrobison', u'name': u'Carly Robison', u'email': u'crobison@caltech.edu'}

Message: SERVER-5781 fixed unittest error code for addFields
Branch: master
https://github.com/mongodb/mongo/commit/4fc451a03f06b696ec563144385aabe2d3d619cf

Comment by Githook User [ 11/Aug/16 ]

Author:

{u'username': u'carlyrobison', u'name': u'Carly Robison', u'email': u'crobison@caltech.edu'}

Message: SERVER-5781 fix error codes for addFields
Branch: master
https://github.com/mongodb/mongo/commit/d1d26b71ca0ac0b20f868dc8accd26769d5fd169

Comment by Githook User [ 11/Aug/16 ]

Author:

{u'username': u'carlyrobison', u'name': u'Carly Robison', u'email': u'crobison@caltech.edu'}

Message: SERVER-5781 addFields stage.

This aggregation stage allows the addition of computed fields to a document while preserving all existing fields.
Branch: master
https://github.com/mongodb/mongo/commit/5a83f956577d1a571ff5acb410aeb06faf5ec92d

Comment by Carly Robison [ 10/Aug/16 ]

We need a page for this new aggregation stage.

Additionally, our courseware (when updated to 3.4) should use $addFields instead of $project:

{<literally every field in the collection>: 1}
Comment by Jose Antonio Illescas Olmo [ 26/Apr/16 ]

I know about $$ROOT.

But this solution required a "custom" JSON structure that I prefer avoid (I like control my JSON structure)

Comment by Asya Kamsky [ 25/Apr/16 ]

Note that there is a limited workaround currently available via $$ROOT system variable. You can preserve incoming document and add other fields, however the original fields and added fields would be on different levels, which is why this feature request exists, it's to be able to add fields to existing document on the same level.

Example:

db.c.find()
{_id:1, name:"Asya"}
db.c.aggregate([ {$project:{doc:"$$ROOT", newfield:{$literal:"NewField"}}} ])
{ "_id" : 1, "doc" : { "_id" : 1, "name" : "Asya" }, "newfield" : "NewField" }

Comment by Jose Antonio Illescas Olmo [ 25/Apr/16 ]

Another Use Case: I define a calculated field "desc2" for every document...

I add "desc" field to any document:

· "desc" sometimes is a simple field or a concatenation of many fields (each with i18n format)
· I use this field ("desc") to show a link description on every document relation (foreign ObjectID) => "Descriptive" links between my documents on HATEOAS application...
· I use this field ("desc") to show a description (not only ObjectIds) to choice between list of items/documents represented as html: select, radio, checkbox

I need add this field without remove the rest of my document fields

Comment by Jose Antonio Illescas Olmo [ 12/Apr/16 ]

I need this feature, currently it's mandatory declare all fields to add new computed fields

note1: in my REST services I provide a calculated i18n description for every document to allows their use on html-select, also I overwrite "_id" as "id" and plainify as string.

note2: another feature is remove fields: all my documents contains "lastModified" date exposed as Last-Modified http-header (coditional get and put) => this field must be removed from document/json content

Thank you

Comment by Thomas Rueckstiess [ 02/Jul/14 ]

I'm voting for this feature, too.

Use Case: I have several $unwind / $group stages in my pipeline, each calculating some values. I'd like to preserve intermediate results all the way to the end of the pipeline, but I don't know which variables those are up front. Therefore, I'd like to have a "results" sub-document that I can just push the results into as sub-documents, and each $project and $group stage would just have to ensure that the "results" sub-document is preserved.

Generated at Thu Feb 08 03:09:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.