[SERVER-41564] Introduce a $replaceFields pipeline stage Created: 06/Jun/19 Updated: 06/Dec/22 |
|
| Status: | Backlog |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Adrian Pinter | Assignee: | Backlog - Query Optimization |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | expression | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Assigned Teams: |
Query Optimization
|
| Participants: |
| Description |
|
The $addFields pipeline stage allows us to replace a field if the field already exists. Unfortunately, the intention to replace rather than add becomes muddled for documents returned from the query that don't contain the field. The nonexistence of the field in those documents changes the effect from replace to add, even though we really just wanted to replace. In my application, I often must return query results where nested arrays are filtered. I've found that using $addFields pipeline stages mostly accomplishes this. For each nested array that I must filter, I construct the corresponding pipeline stage as follows:
But a problem arises when one of the path segments in arrayFilter.field doesn't exist in the database. The problem is compounded when arrayFilter.field is nested, i.e. contains dots. Under those circumstances, I want the stage to have zero effect. Unfortunately, the query result is such that:
Such problems could be avoided by introducing a $replaceFields pipeline stage that we can use to unambiguously declare our intention to replace rather than add. |
| Comments |
| Comment by Eric Sedor [ 10/Jun/19 ] | |||
|
Thanks for the added detail, HappyNomad; I'm passing this on to an appropriate team for consideration. Please keep an eye on this ticket in case we have more questions or additional information about currently available workarounds. | |||
| Comment by Adrian Pinter [ 10/Jun/19 ] | |||
|
As an alternative to the $replaceFields feature idea, you could consider changing $$REMOVE's behavior in situations like this. It seems unhelpful to remove only the leaf field, while leaving behind the trail of empty documents that lead to it. $$REMOVE could instead remove (or never include in the first place) every document in the path that doesn't exist in the database. | |||
| Comment by Adrian Pinter [ 08/Jun/19 ] | |||
|
Hi @eric.sedor, thanks for the doc links. Including computed fields was what I was already doing, but the "conditionally exclude fields" link led me to try something new:
That got me a little closer, but no cigar. The problem remains when arrayFilter.field is nested, i.e. contains dots. In that case, the path to the nonexistent array is still materialized in the query results. The only improvement over my earlier attempt is that the leaf array field is now absent rather than present with a null value. | |||
| Comment by Eric Sedor [ 06/Jun/19 ] | |||
|
Hi HappyNomad; Do the $project options to conditionally exclude fields and include computed fields help satisfy your use-case? |