[SERVER-72651] $match filter is erroneously pushed past $project into COLLSCAN Created: 09/Jan/23  Updated: 29/Oct/23  Resolved: 19/Jan/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 6.0.3
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Steve Tarzia Assignee: Steve Tarzia
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on SERVER-67416 Always sort fieldpath string sets usi... Closed
Backwards Compatibility: Fully Compatible
Operating System: ALL
Steps To Reproduce:

On v6.0:

const c = db.bf27387;
c.drop();
c.save({_id: 0, array: ["abc"]});
 
const res = c.aggregate(
    [{$project: {"obj.obj.array": 1}},
    {$match: {$expr: {$getField: {$literal: "array"}}}}]);
 
// The bug caused the query above to return { "_id" : 0 } instead of no documents.
assert.eq(0, res.toArray().length);

Mongo v6.0 will erroneously generate the following plan for the aggregation:

                "winningPlan" : {
                        "stage" : "PROJECTION_DEFAULT",
                        "transformBy" : {
                                "_id" : true,
                                "obj" : {
                                        "obj" : {
                                                "array" : true
                                        }
                                }
                        },
                        "inputStage" : {
                                "stage" : "COLLSCAN",
                                "filter" : {
                                        "$expr" : {
                                                "$getField" : {
                                                        "field" : {
                                                                "$const" : "array"
                                                        },
                                                        "input" : "$$CURRENT"
                                                }
                                        }
                                },
                                "direction" : "forward"
                        }
                },

The filter is merged into the collscan, putting the filter before the projection. This is problematic because the semantics of the filter are different when operating on the full (unprojected) document.

The correct plan, produced by v6.1, is:

{
	"explainVersion" : "1",
	"stages" : [
		{
			"$cursor" : {
				"queryPlanner" : {
					"namespace" : "test.bf27387",
					"indexFilterSet" : false,
					"parsedQuery" : {
						
					},
					"queryHash" : "92148A8F",
					"planCacheKey" : "92148A8F",
					"maxIndexedOrSolutionsReached" : false,
					"maxIndexedAndSolutionsReached" : false,
					"maxScansToExplodeReached" : false,
					"winningPlan" : {
						"stage" : "PROJECTION_DEFAULT",
						"transformBy" : {
							"_id" : true,
							"obj" : {
								"obj" : {
									"array" : true
								}
							}
						},
						"inputStage" : {
							"stage" : "COLLSCAN",
							"direction" : "forward"
						}
					},
					"rejectedPlans" : [ ]
				}
			}
		},
		{
			"$match" : {
				"$expr" : {
					"$getField" : {
						"field" : {
							"$const" : "array"
						},
						"input" : "$$CURRENT"
					}
				}
			}
		}
	],...} 

Sprint: QE 2023-01-23
Participants:
Linked BF Score: 120

 Description   

A $match filter can be pushed past a $project into a COLLSCAN only if the $project has no impact on the filter results.  For this to be true, the $match filters' paths must be a subset of the $project paths.  This check seems to be broken in v6.0.

This ticket came out of rui.liu@mongodb.com's investigation of BF-27387.

This issue is only seen in v6.0 because it was fixed for 6.1.0-rc0 by a change to isIndependentOf in SERVER-67416.  To fix this bug in v6.0 we can try to backport SERVER-67416.  This ticket may also track work to add a js test for this bug.

v5.0 does not exhibit this bug.



 Comments   
Comment by Githook User [ 19/Jan/23 ]

Author:

{'name': 'Steve Tarzia', 'email': 'steve.tarzia@mongodb.com', 'username': 'starzia'}

Message: SERVER-72651 test $match filter erroneously pushed past $project
Branch: master
https://github.com/mongodb/mongo/commit/8d6a434730b821ed8256b687ae3c0ebd8777b0cf

Comment by Steve Tarzia [ 18/Jan/23 ]

a more minimal repro of the issue is:

const c = db.server72651;
c.drop();
 
assert.commandWorked(c.insert({_id: 0, a: 1}));
// The bug caused the query below to return {"_id" : 0} instead of no documents.
assert.eq(
    [],
    c.aggregate([{$project: {"b": 1}}, {$match: {$expr: {$getField: {$literal: "a"}}}}]).toArray());

Comment by Steve Tarzia [ 13/Jan/23 ]

I verified that v5.0 does not exhibit this bug.

Generated at Thu Feb 08 06:22:25 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.