[SERVER-31786] $concatArrays not working on existing multi array field Created: 01/Nov/17 Updated: 27/Oct/23 Resolved: 10/Nov/17 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | Aggregation Framework |
| Affects Version/s: | 3.4.6 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Joel Goldfinger | Assignee: | David Storch |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Operating System: | ALL | ||||||||
| Steps To Reproduce: | Given a collection:
Result should be:
Result is:
|
||||||||
| Sprint: | Query 2017-12-04 | ||||||||
| Participants: | |||||||||
| Description |
|
The $concatArrays operator does not work when using a multi array field already in the collection. The current work around is to use $reduce with $concatArrays. |
| Comments |
| Comment by David Storch [ 15/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi devnopt, That makes sense. Would you mind filing a SERVER ticket of type "New Feature" describing your use case and the desired addition? The MongoDB Query Team can carry this process through our usual planning and prioritization process. This is unlikely to be something that we would schedule to be built in the immediate future, but it would be good to record it on our backlog as future work. Best, | |||||||||||||||||||||||||||||||||||||||||
| Comment by Joel Goldfinger [ 13/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Dave, That makes sense as to why it is working the way it does, but it leaves a major gap in how to handle pre-existing fields in MongoDB. Using reduce has a significant overhead as it is merging the new array for each inner array. Would it be possible to allow building a custom ExpressionNary that can then be used? For this example lets call it $expressionNary, which it takes <expression> and creates an ExpressionNary.
Result is:
Thanks. | |||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 13/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi devnopt, this is a peculiarity of the expression language, but it is working as intended. There are a class of expressions which accept multiple arguments, expressed as an array. (In the code, these inherit from ExpressionNary.) For any such n-ary expression, which I will call $nary, the syntax {$nary: "$myArg"} is simply a shorthand for {$nary: ["$myArg"]}. These two expression are identical in meaning. The same is not true for expressions that take named arguments, as these are not n-ary expressions. $map and $filter are examples of expressions that take named arguments. In these cases, an expression like ["$myExpr"] is interpreted as an ExpressionArray---that is, an expression which evaluates to an array, where each element of the array is the result of evaluating a subexpression. For instance, suppose "$a" evalutes to 1 and "$b" evaluates to [2, 3]. In that case the ExpressionArray
will evaluate to
It is ExpressionArray evaluation which leads to the additional array wrapping in the result set for your example queries. I hope this helps to clarify! Best, | |||||||||||||||||||||||||||||||||||||||||
| Comment by Joel Goldfinger [ 10/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Dave, The $concatArrays expression will convert { $concatArrays: "$items" } to { $concatArrays: ["$items"] } where as the named subexpression "input" for $map takes the variable as is. Why should $concatArrays and $map behave differently with respect to how they resolve the variable? See example below.
Example 1A and 1B both result in:
Example 2A and 2B are different and result in:
Thanks. | |||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 10/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi devnopt, I'm not sure I understand. Can you explain what you mean by "automatically wrap expression #1 in an array"? $filter and $map are in a different category from $concatArrays and $setUnion. Rather than being a variable arity expression that accepts either a single subexpression or an array of subexpressions, $filter and $map always have named subexpressions like "input", "as", "in", and "cond". Best, | |||||||||||||||||||||||||||||||||||||||||
| Comment by Joel Goldfinger [ 10/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Dave, The issue with this is that it isn't consistent with the way arrays are handled for other operations. Some array operations such as $concatArrays and $setUnion automatically wrap expression #1 in an array whereas others such as $filter and $map do not. It should be up to the user to determine whether to wrap the field in an array. Thanks. | |||||||||||||||||||||||||||||||||||||||||
| Comment by David Storch [ 10/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hi devnopt, This is quite subtle, but the aggregation system is actually working as designed. The $concatArrays, like all of the variable arity aggregation expressions, may take one or more arguments. Consider a generic variable arity expression, $varargs. Such an expression is allowed to be invoked with a single argument:
It can also be invoked with multiple arguments, in which case the arguments are passed as an array:
This means that if the input document is {myArg: [[1, 2, 3], [4, 5, 6]]}, the following aggregation expressions actually mean very different things:
The former means to call $varargs with the single argument [[1, 2, 3], [4, 5, 6]]. The latter means to call $varargs with two arguments, [1, 2, 3] and [4, 5, 6]. In the case that $varargs is $concatArrays, [[1, 2, 3], [4, 5, 6]] concatenated with nothing else should simply be a no-op. On the other hand, [1, 2, 3] concatenated with [4, 5, 6] should return [1, 2, 3, 4, 5, 6]. Your approach of using $concatArrays with $reduce sounds correct to me. I'm closing this ticket as Works as Designed. Please feel free to reach out on this ticket with any questions or concerns! Best, | |||||||||||||||||||||||||||||||||||||||||
| Comment by Mark Agarunov [ 09/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hello devnopt Thank you for the additional infomration. I've set the fixVersion on this ticket to "Needs Triage" for this to be scheduled against our currently planned work. Updates will be posted on this ticket as they happen. Thanks, | |||||||||||||||||||||||||||||||||||||||||
| Comment by Joel Goldfinger [ 08/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
It appears this issue also applies to $setUnion. May want to look at the set operations too. Thanks. | |||||||||||||||||||||||||||||||||||||||||
| Comment by Mark Agarunov [ 06/Nov/17 ] | |||||||||||||||||||||||||||||||||||||||||
|
Hello devnopt, Thank you for the report. I've confirmed this behavior and believe it may be caused by the $items field internally being nested into an array since the parser sees only a single element. We are still investigating this issue and will provide updates on this ticket as they are available. Thanks, |