[SERVER-41564] Introduce a $replaceFields pipeline stage Created: 06/Jun/19  Updated: 06/Dec/22

Status: Backlog
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Adrian Pinter Assignee: Backlog - Query Optimization
Resolution: Unresolved Votes: 0
Labels: expression
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Assigned Teams:
Query Optimization
Participants:

 Description   

The $addFields pipeline stage allows us to replace a field if the field already exists.  Unfortunately, the intention to replace rather than add becomes muddled for documents returned from the query that don't contain the field.  The nonexistence of the field in those documents changes the effect from replace to add, even though we really just wanted to replace.

In my application, I often must return query results where nested arrays are filtered.  I've found that using $addFields pipeline stages mostly accomplishes this.  For each nested array that I must filter, I construct the corresponding pipeline stage as follows:

"{ $addFields: { '" + arrayFilter.field +
    "': { $filter: { input: '$" + arrayFilter.field +
    "', cond: { " + arrayFilter.cond + " } } } } }" );

But a problem arises when one of the path segments in arrayFilter.field doesn't exist in the database.  The problem is compounded when arrayFilter.field is nested, i.e. contains dots.

Under those circumstances, I want the stage to have zero effect. Unfortunately, the query result is such that:

  1. the path leading to the nonexistent array is materialized
  2. the unwanted array is given a null value.

Such problems could be avoided by introducing a $replaceFields pipeline stage that we can use to unambiguously declare our intention to replace rather than add.



 Comments   
Comment by Eric Sedor [ 10/Jun/19 ]

Thanks for the added detail, HappyNomad; I'm passing this on to an appropriate team for consideration. Please keep an eye on this ticket in case we have more questions or additional information about currently available workarounds.

Comment by Adrian Pinter [ 10/Jun/19 ]

As an alternative to the $replaceFields feature idea, you could consider changing $$REMOVE's behavior in situations like this.  It seems unhelpful to remove only the leaf field, while leaving behind the trail of empty documents that lead to it.  $$REMOVE could instead remove (or never include in the first place) every document in the path that doesn't exist in the database.

Comment by Adrian Pinter [ 08/Jun/19 ]

Hi @eric.sedor, thanks for the doc links.  Including computed fields was what I was already doing, but the "conditionally exclude fields" link led me to try something new:

"{ $addFields: { '" + arrayFilter.field +
	"': { $ifNull: [ { $filter: { input: '$" + arrayFilter.field +
	"', cond: { " + arrayFilter.cond + " } } }, '$$REMOVE' ] } } }" );

That got me a little closer, but no cigar.  The problem remains when arrayFilter.field is nested, i.e. contains dots.  In that case, the path to the nonexistent array is still materialized in the query results.  The only improvement over my earlier attempt is that the leaf array field is now absent rather than present with a null value.

Comment by Eric Sedor [ 06/Jun/19 ]

Hi HappyNomad; Do the $project options to conditionally exclude fields and include computed fields help satisfy your use-case?

Generated at Thu Feb 08 04:58:05 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.