[SERVER-31696] Aggregation's renamed fields analysis is broken for dotted field paths, causes incorrect query results Created: 24/Oct/17  Updated: 30/Oct/23  Resolved: 04/Nov/17

Status: Closed
Project: Core Server
Component/s: Aggregation Framework
Affects Version/s: 3.6.0-rc0
Fix Version/s: 3.6.0-rc3

Type: Bug Priority: Critical - P2
Reporter: David Storch Assignee: Justin Seyster
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
is caused by SERVER-27115 Track fields renamed by $project in a... Closed
Related
related to SERVER-32690 Aggregation can trip invariant relate... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Sprint: Query 2017-11-13
Participants:

 Description   

The optimization added in SERVER-27115 to allow index usage in the case of a renamed field is broken for matches on a dotted field path. See the example below. This is a regression introduced during 3.5 development. The 3.6 release candidates are affected, but no pre-existing stable release series is affected.

> db.c.drop();
true
> db.c.insert({x: [{y: 1}, {y: 2}]});
WriteResult({ "nInserted" : 1 })
 
// Match with no dotted field path returns the correct result. The explain indicates that
// the $match stage swapped in front of the $addFields, and was absorbed into the
// $cursor stage after being correctly renamed to a match on "x".
> db.c.aggregate([{$addFields: {z: "$x"}}, {$match: {z: {y: 1}}}]);
{ "_id" : ObjectId("59ef5d62b49385b9971e2c09"), "x" : [ { "y" : 1 }, { "y" : 2 } ], "z" : [ { "y" : 1 }, { "y" : 2 } ] }
> db.c.explain().aggregate([{$addFields: {z: "$x"}}, {$match: {z: {y: 1}}}]);
{
	"stages" : [
		{
			"$cursor" : {
				"query" : {
					"x" : {
						"$eq" : {
							"y" : 1
						}
					}
				},
				"queryPlanner" : {
					"plannerVersion" : 1,
					"namespace" : "test.c",
					"indexFilterSet" : false,
					"parsedQuery" : {
						"x" : {
							"$eq" : {
								"y" : 1
							}
						}
					},
					"winningPlan" : {
						"stage" : "COLLSCAN",
						"filter" : {
							"x" : {
								"$eq" : {
									"y" : 1
								}
							}
						},
						"direction" : "forward"
					},
					"rejectedPlans" : [ ]
				}
			}
		},
		{
			"$addFields" : {
				"z" : "$x"
			}
		}
	],
	"ok" : 1
}
 
// When a similar match is expressed using the dotted path "z.y", the matching document
// is incorrectly omitted from the result set. The explain indicates that the rename to "x.y"
// was not applied correctly.
> db.c.explain().aggregate([{$addFields: {z: "$x"}}, {$match: {"z.y": 1}}]);
{
	"stages" : [
		{
			"$cursor" : {
				"query" : {
					"z.y" : {
						"$eq" : 1
					}
				},
				"queryPlanner" : {
					"plannerVersion" : 1,
					"namespace" : "test.c",
					"indexFilterSet" : false,
					"parsedQuery" : {
						"z.y" : {
							"$eq" : 1
						}
					},
					"winningPlan" : {
						"stage" : "COLLSCAN",
						"filter" : {
							"z.y" : {
								"$eq" : 1
							}
						},
						"direction" : "forward"
					},
					"rejectedPlans" : [ ]
				}
			}
		},
		{
			"$addFields" : {
				"z" : "$x"
			}
		}
	],
	"ok" : 1
}



 Comments   
Comment by Githook User [ 04/Nov/17 ]

Author:

{'name': 'Justin Seyster', 'username': 'jseyster', 'email': 'justin.seyster@mongodb.com'}

Message: SERVER-31696 Rename path prefixes in applyRenamesToExpression().

Previously, applyRenamesToExpression() would only rename a path if the
entire path matched the rename rule.
Branch: master
https://github.com/mongodb/mongo/commit/d12505f5545b9109c12cb472f01c4879f814b453

Comment by David Storch [ 24/Oct/17 ]

justin.seyster, this should be your highest priority task. Since this is a regression affecting the correctness of query results, I think it needs to be fixed in a 3.6.0 release candidate. Let's target RC2.

The short version of the problem is that the code which applies renamed fields to match expressions being swapped forward contains a mapping like "z" => "x", which means that a stored field named "x" has been renamed to "z". When swapping a match expression forwards, we look for matches on the path "z" and rename them to "x":

https://github.com/mongodb/mongo/blob/43b018194e1a49249092b871f3b7396e473d1426/src/mongo/db/matcher/expression_algo.cpp#L279-L296

However, we do not have the logic to correctly rename a dotted path like "z.y" to "x.y". This needs to be implemented and tested.

Generated at Thu Feb 08 04:27:54 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.