[SERVER-79205] [CQF] $not $eq array, on a missing field, should return true Created: 21/Jul/23 Updated: 22/Sep/23 |
|
| Status: | Open |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | David Percy | Assignee: | Backlog - Query Optimization |
| Resolution: | Unresolved | Votes: | 0 |
| Labels: | bonsai-semantic-difference | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Assigned Teams: |
Query Optimization
|
||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Sprint: | QO 2023-08-21 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
This is something the fuzzer discovered. Example collection and queries on classic:
Same example with forceBonsai:
This only happens when the constant is an array. For example this query is correct:
The Bonsai plan before optimization is:
and after optimization:
It looks like the relevant rewrites are:
This must be happening due to confusion around Nothing vs False. Missing fields are represented as Nothing, and most operations on Nothing return Nothing. I think even UnaryOp Not preserves Nothing. But at some point we coerce Nothing to False: this either happens in the Filter stage or in the EvalFilter expression. I would probably first try disabling the above rewrites and see whether that makes the result correct. Then we can decide what changes are necessary to the rewrites, or to the initial ABT translation. |
| Comments |
| Comment by Hana Pearlman [ 31/Jul/23 ] | ||||||||||||||||||||
|
The initial ABT translation looks correct to me. It's the EvalFilter expression that coerces Nothing to False, so in the translated ABT the path below the EvalFilter will return Nothing, the EvalFilter will coerce this to False, and the UnaryOp [Not] will return True. The same applies to the first Filter node in the ABT post-optimization. The second Filter node looks like the problem to me. I expect that the PathCompare [Neq] returns Nothing, which the EvalFilter coerces to False. I am not very familiar with the NotPushdown rewrites, but it seems to me that the "combining Not Eq into Neq" step is causing the issue. UnaryOp [Not] above EvalFilter with PathCompare [eq] returns True when the path returns Nothing. EvalFilter with PathCompare [neq] returns False when the path returns Nothing. It doesn't seem to me that this issue is specific to arrays, even though we are only seeing it with array comparisons currently. (edit: woops, that's what the last comment shows) | ||||||||||||||||||||
| Comment by David Percy [ 21/Jul/23 ] | ||||||||||||||||||||
|
I found one more way to trigger this:
This example also involves combining Not Eq into Neq. That rewrite can only happen when there's no implicit array traversal, which is why the rewrite only happen when the index exists--the index metadata lets us eliminate the array traversal. This rewrite also happens with whole-array comparison: one of the two disjuncts doesn't have any implicit array traversal. |