[SERVER-63030] Correlated sub-pipeline analysis is incorrect for $unionWith, leading to incorrect results Created: 27/Jan/22  Updated: 10/Feb/22  Resolved: 10/Feb/22

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.0.5, 4.4.12, 5.2.0-rc6
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: David Storch Assignee: Alya Berciu
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to SERVER-63141 Difference in $lookup/$redact/$let be... Closed
Operating System: ALL
Steps To Reproduce:

The following script reproduces the bug:

(function() {
"use strict";
 
const localColl = db.local_coll;
localColl.drop();
 
assert.commandWorked(localColl.insert({a: "notMagic"}));
assert.commandWorked(localColl.insert({a: "magicConstant"}));
 
const firstUnionColl = db.first_union_coll;
firstUnionColl.drop();
assert.commandWorked(firstUnionColl.insert({unionedData: 1}));
 
const secondUnionColl = db.second_union_coll;
secondUnionColl.drop();
assert.commandWorked(secondUnionColl.insert({unionedData: 2}));
 
let results = localColl.aggregate([
    {$lookup: {
        from: firstUnionColl.getName(),
        as: "as",
        let: {correlatedVar: "$a"},
        pipeline: [
            {$_internalInhibitOptimization: {}},
            {$unionWith: {
                coll: secondUnionColl.getName(),
                pipeline: [
                    {$match: {$expr: {$eq: ["$$correlatedVar", "magicConstant"]}}}
                ]
            }}
        ]
    }}
]).toArray();
 
printjson(results);
}());

Sprint: QO 2022-02-21
Participants:

 Description   

The $lookup stage has some analysis to try to identify a non-correlated prefix of its sub-pipeline. It such a prefix is found, then it can be executed just once and the results materialized, in order to avoid repeated execution of the non-correlated portion of the query.

For this optimization to work, the implementation relies on the getDependencies() implementation of each DocumentSource to correctly report all of its variable references. This allows the code looking for a non-correlated prefix to check whether any correlated variables are used. However, DocumentSourceUnionWith::getDependencies() is unimplemented right now. Therefore, it fails to report any of the variable references inside its sub-pipeline.

As a result, if there is a $lookup with a $unionWith inside of it, and the $unionWith's sub-pipeline has a correlated variable reference, then the correlation analysis will incorrectly identify the $unionWith stage as non-correlated. Materialization of a supposedly non-correlated pipeline prefix will kick in, resulting in incorrect query results. The attached repro script has a concrete example of this situation.



 Comments   
Comment by Alya Berciu [ 10/Feb/22 ]

Closing as duplicate since SERVER-63141 and associated backports are handling the fix.

Comment by Alya Berciu [ 04/Feb/22 ]

Looks like the fix for SERVER-63141 should also address this bug.

Generated at Thu Feb 08 05:56:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.