[SERVER-51303] Lookup stage followed by $match on type uses wrong field in matching condition Created: 02/Oct/20 Updated: 29/Oct/23 Resolved: 10/Nov/20 |
|
| Status: | Closed |
| Project: | Core Server |
| Component/s: | None |
| Affects Version/s: | 4.2.9, 4.4.1, 3.6.20, 4.0.21 |
| Fix Version/s: | 4.8.1, 4.4.2, 4.2.11, 3.6.21, 4.0.22 |
| Type: | Bug | Priority: | Blocker - P1 |
| Reporter: | Thomas Rueckstiess | Assignee: | Jacob Evans |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | qopt-team | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Backwards Compatibility: | Fully Compatible | ||||||||||||||||
| Operating System: | ALL | ||||||||||||||||
| Backport Requested: |
v4.4, v4.2, v4.0, v3.6
|
||||||||||||||||
| Steps To Reproduce: | See below. |
||||||||||||||||
| Sprint: | Query 2020-10-19, Query 2020-11-02, Query 2020-11-16 | ||||||||||||||||
| Participants: | |||||||||||||||||
| Description |
|
Issue Status as of Nov 12, 2020 ISSUE DESCRIPTION AND IMPACT Matching by $type on a looked up field after a $lookup stage in an aggregation operation leads to incorrect results, because the $match on $type is pushed into the $lookup without removing the reference to the new field created by $lookup. This failure is unique to the $type operator. For example, the following operation matches on "lookedup.name" after a $lookup into the "lookedup" field:
MongoDB incorrectly optimizes the aggregation pipeline by attempting to match on "lookedup.name" in the "second" collection, where "lookedup.name" does not exist. The correct behavior is for the optimized lookup stage to match on “name” instead. DIAGNOSIS AND AFFECTED VERSIONS All versions since 3.4.0 are affected by this issue. Running the .explain() method will help identify the incorrect matching stage.
If the query optimizer moves the $match stage into the $lookup stage while still referencing the field created by $lookup, the operation is impacted. For example:
where
should be
REMEDIATION AND WORKAROUNDS
Original DescriptionA $lookup stage followed by a $match on the type of a looked up field does not work. It seems to be using the wrong field name when the $match gets pushed inside the $lookup stage. Steps to Reproduce1. Create two collections as follows
2. Run the following aggregation on collection "first"
3. In the explain, you can see that the $match gets pushed into the lookup stage but uses the original field name "lookedup.name" which doesn't exist in the remote collection.
If a non-$type query is used (e.g. equality match), the field instead is just "name", which is the correct behavior.
Expected ResultsAggregation returns the document because the field lookedup.name is a string Actual ResultsAggregation returns no documents.
—
Hat tip to nathan.smyth who discovered this bug. |
| Comments |
| Comment by Githook User [ 10/Nov/20 ] |
|
Author: {'name': 'Jacob Evans', 'email': 'jacob.evans@10gen.com'}Message: |
| Comment by Githook User [ 10/Nov/20 ] |
|
Author: {'name': 'Jacob Evans', 'email': 'jacob.evans@10gen.com'}Message: |
| Comment by Githook User [ 10/Nov/20 ] |
|
Author: {'name': 'Jacob Evans', 'email': 'jacob.evans@10gen.com'}Message: |
| Comment by Githook User [ 10/Nov/20 ] |
|
Author: {'name': 'Jacob Evans', 'email': 'jacob.evans@10gen.com'}Message: |
| Comment by Githook User [ 10/Nov/20 ] |
|
Author: {'name': 'Jacob Evans', 'email': 'jacob.evans@10gen.com'}Message: |
| Comment by David Storch [ 04/Nov/20 ] |
|
I did a bit of digging into the origin of this bug, and it appears that it was introduced with If my archaeology in the git history is correct, this check was not introduced by following some subtle line of reasoning around the semantics of the $type match expression, but rather was added for a more pedestrian reason. The original code from |