[CSHARP-2723] Select() inside Project() does not properly reference the Project lambda variable Created: 31/Aug/19  Updated: 28/Oct/23  Resolved: 13/Oct/21

Status: Closed
Project: C# Driver
Component/s: Linq
Affects Version/s: None
Fix Version/s: 2.14.0

Type: Bug Priority: Major - P3
Reporter: Richard Collette Assignee: Robert Stam
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Zip Archive MongoDbTesting.zip     File archive.gz    
Issue Links:
Depends
depends on CSHARP-3314 LINQ3: Implement Known Serializers st... Closed
Epic Link: CSHARP-3615

 Description   

When a Select() is used inside a Project(), and the Select() expression refers to the Project() lambda variable, the generated projection does not properly reference the Project() lambda variable and its properties.

Given a local database named "test" and

db.parents.insert([
    {_id:1, name:"parent1"},
    {_id:2, name:"parent2"}
]);
 
db.children.insert([
    {_id:1, name:"child1", parentId:2},
    {_id:2, name:"child2", parentId:1}
]);
 
 
db.grandChildren.insert([
    {_id:1, name:"grandchild1", childId: 2},
    {_id:2, name:"grandchild2", childId: 1}
]);

And query method GetTree()

using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using MongoDB.Bson;
using MongoDB.Driver;
 
namespace MongoDbTest.Repository
{
    public class Repository : IRepository
    {
        private readonly IMongoCollection<Child> _children;
        private readonly IMongoCollection<GrandChild> _grandChildren;
        private readonly IMongoCollection<Parent> _parents;
 
        // There is no way to refer to a let variable ($$products) from a lambda function.
        // We have to use a BsonDocument (strings unfortunately) to define our lookup pipeline.
        // In reality, there are other criteria in the $and
        private const string DeliveriesLookupPipelineString = @"
{$match:{
  $expr:{
    $and:[
      {$in:[""$childId"",""$$children._id""]}
     ]
  }
 }
}
";
 
        private static readonly PipelineDefinition<GrandChild, GrandChild> GrandchildrenLookupPipeline = new[]
        {
            BsonDocument.Parse(DeliveriesLookupPipelineString)
        };
 
        public Repository(
            IMongoCollection<Parent> parents,
            IMongoCollection<Child> children,
            IMongoCollection<GrandChild> grandChildren)
        {
            _parents = parents;
            _children = children;
            _grandChildren = grandChildren;
        }
 
        public Task<List<ParentTree>> GetTree()
        {
            return _parents
                .Aggregate()
                .Lookup<Parent, Child, ParentProjection>(
                    _children,
                    parent => parent.Id,
                    child => child.ParentId,
                    parentProjection => parentProjection.Children)
                .Lookup<ParentProjection, GrandChild, GrandChild, IEnumerable<GrandChild>, ParentProjection>(
                    _grandChildren,
                    new BsonDocument {{"children", "$children"}},
                    GrandchildrenLookupPipeline,
                    childProjection => childProjection.GrandChildren)
                .Project(parent => new ParentTree
                {
                    Id = parent.Id,
                    ParentName = parent.Name,
                    Children = parent.Children
                        .Select(child => new ChildTree
                        {
                            Id = child.Id,
                            Name = child.Name,
                            GrandChildren = parent
                                .GrandChildren // For some reason the generated code refers to child.grandChildren
                                .Where(gc =>
                                    gc.ChildId ==
                                    child.Id) // and here the generated code compares gc.childId to gc._id rather than to child._id
                                .Select(gc =>
                                    new GrandChildProjection()
                                    {
                                        Id = gc.Id,
                                        Name = gc.Name
                                    })
                        })
                }).ToListAsync();
        }
    }
}

The following MongoDb aggregate is generated (2 lines commented with // incorrect)

db.parents.aggregate(
    [
   {
      "$lookup":{
         "from":"children",
         "localField":"_id",
         "foreignField":"parentId",
         "as":"children"
      }
   },
   {
      "$lookup":{
         "from":"grandChildren",
         "let":{
            "children":"$children"
         },
         "pipeline":[
            {
               "$match":{
                  "$expr":{
                     "$and":[
                        {
                           "$in":[
                              "$childId",
                              "$$children._id"
                           ]
                        }
                     ]
                  }
               }
            }
         ],
         "as":"grandChildren"
      }
   },
   {
      "$project":{
         "Id":"$_id",
         "ParentName":"$name",
         "Children":{
            "$map":{
               "input":"$children",
               "as":"child",
               "in":{
                  "Id":"$$child._id",
                  "Name":"$$child.name",
                  "GrandChildren":{
                     "$map":{
                        "input":{
                           "$filter":{
                              "input":"$$child.grandChildren", // This is wrong
                              "as":"gc",
                              "cond":{
                                 "$eq":[
                                    "$$gc.childId",
                                    "$$gc._id" // This is wrong
                                 ]
                              }
                           }
                        },
                        "as":"gc",
                        "in":{
                           "Id":"$$gc._id",
                           "Name":"$$gc.name"
                        }
                     }
                  }
               }
            }
         },
         "_id":0
      }
   }
]
);

But the proper aggregate would be (2 lines commented with // correct)

db.parents.aggregate(
    [
   {
      "$lookup":{
         "from":"children",
         "localField":"_id",
         "foreignField":"parentId",
         "as":"children"
      }
   },
   {
      "$lookup":{
         "from":"grandChildren",
         "let":{
            "children":"$children"
         },
         "pipeline":[
            {
               "$match":{
                  "$expr":{
                     "$and":[
                        {
                           "$in":[
                              "$childId",
                              "$$children._id"
                           ]
                        }
                     ]
                  }
               }
            }
         ],
         "as":"grandChildren"
      }
   },
   {
      "$project":{
         "Id":"$_id",
         "ParentName":"$name",
         "Children":{
            "$map":{
               "input":"$children",
               "as":"child",
               "in":{
                  "Id":"$$child._id",
                  "Name":"$$child.name",
                  "GrandChildren":{
                     "$map":{
                        "input":{
                           "$filter":{
                              "input":"$grandChildren", // correct
                              "as":"gc",
                              "cond":{
                                 "$eq":[
                                    "$$gc.childId",
                                    "$$child._id" // correct
                                 ]
                              }
                           }
                        },
                        "as":"gc",
                        "in":{
                           "Id":"$$gc._id",
                           "Name":"$$gc.name"
                        }
                     }
                  }
               }
            }
         },
         "_id":0
      }
   }
]
);

Attached is the .NET Core solution, and database archive



 Comments   
Comment by Robert Stam [ 13/Oct/21 ]

This issue has been fixed in the new LINQ provider (known as LINQ3) which will be included in the upcoming 2.14 release.

Configure your MongoClientSettings to use LinqProvider.V3 if you want to use this functionality.

To configure a client to use the LINQ3 provider use code like the following

var connectionString = "mongodb://localhost";
var clientSettings = MongoClientSettings.FromConnectionString(connectionString);
clientSettings.LinqProvider = LinqProvider.V3;
var client = new MongoClient(clientSettings);

Comment by Githook User [ 13/Oct/21 ]

Author:

{'name': 'rstam', 'email': 'robert@robertstam.org', 'username': 'rstam'}

Message: CSHARP-2723: Refactor Linq3TestHelpers Translate method to not use Stages property.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/9e8e52540e7c131247b7050cee2d4ed0c71c074c

Comment by Githook User [ 13/Oct/21 ]

Author:

{'name': 'rstam', 'email': 'robert@robertstam.org', 'username': 'rstam'}

Message: CSHARP-2723: Verify that this scenario works in LINQ3.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/0a210b504829e0cc38225f7fcf2eed0af898305d

Comment by Robert Stam [ 24/Sep/21 ]

With one minor change to the LINQ3 implementation I was able to get this scenario working using LINQ3.

Comment by Robert Stam [ 24/Sep/21 ]

https://github.com/rstam/mongo-csharp-driver/pull/194

Comment by Richard Collette [ 02/Sep/19 ]

I attempting to debug, this but having difficult for a couple reasons.  

It seems the dotnet debugger is crashing (I'm using rider on a Mac).  And the code inside AggregateProjectTranslator is recursive (maybe?).  It takes a lot to step through it.

What I can see definitively at the moment is that in AggregateProjectTranslator, the returned Expression from PipelineBindingContext.Bind() incorrectly maps the "child" lambda parameter to document

new ParentTree() {Children = {document}{children}.Select(new ChildTree() {GrandChildren = {document}{grandChildren}.Where(({document}{childId} == {document}{_id})).Select(value(MongoDbTest.Repository.GrandChildProjection))})}

The debugger crash I am getting is (and I'm not saying this is a driver issue necessarily):

Process:               dotnet [17273]
Path:                  /usr/local/share/dotnet/dotnet
Identifier:            dotnet
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        mono-sgen [17265]
Responsible:           dotnet [17273]
User ID:               502
 
Date/Time:             2019-09-02 12:02:18.993 -0400
OS Version:            Mac OS X 10.14.6 (18G87)
Report Version:        12
Anonymous UUID:        237AA4CB-5DF1-0F4F-B3EA-9EC42ECC8B26
 
Sleep/Wake UUID:       AC2D0D1B-712D-4F89-8820-A7FA32E57014
 
Time Awake Since Boot: 370000 seconds
Time Since Wake:       5800 seconds
 
System Integrity Protection: enabled
 
Crashed Thread:        20
 
Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x000000000000000b
Exception Note:        EXC_CORPSE_NOTIFY
 
Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [17273]
 
 
Thread 20 Crashed:
0   libcoreclr.dylib              	0x000000010405fa8e ILStubManager::TraceManager(Thread*, TraceDestination*, _CONTEXT*, unsigned char**) + 206
1   libcoreclr.dylib              	0x0000000103fda10c EEDbgInterfaceImpl::TraceManager(Thread*, StubManager*, TraceDestination*, _CONTEXT*, unsigned char**) + 172
2   libcoreclr.dylib              	0x0000000103f25d76 DebuggerStepper::TriggerPatch(DebuggerControllerPatch*, Thread*, TRIGGER_WHY) + 678
3   libcoreclr.dylib              	0x0000000103f20fdd DebuggerController::ScanForTriggers(unsigned char const*, Thread*, _CONTEXT*, DebuggerControllerQueue*, SCAN_TRIGGER, TP_RESULT*) + 749
4   libcoreclr.dylib              	0x0000000103f21694 DebuggerController::DispatchPatchOrSingleStep(Thread*, _CONTEXT*, unsigned char const*, SCAN_TRIGGER) + 116
5   libcoreclr.dylib              	0x0000000103f22a57 DebuggerController::DispatchNativeException(_EXCEPTION_RECORD*, _CONTEXT*, unsigned int, Thread*) + 551
6   libcoreclr.dylib              	0x0000000103f30886 Debugger::FirstChanceNativeException(_EXCEPTION_RECORD*, _CONTEXT*, unsigned int, Thread*) + 182
7   libcoreclr.dylib              	0x00000001041c2823 HandleHardwareException(PAL_SEHException*) + 131
8   libcoreclr.dylib              	0x0000000103ea9c11 SEHProcessException(PAL_SEHException*) + 337
9   libcoreclr.dylib              	0x0000000103ee7ac5 PAL_DispatchException + 181
10  libcoreclr.dylib              	0x0000000103ee7666 PAL_DispatchExceptionWrapper + 10
...
...
...
105 libcoreclr.dylib              	0x000000010424d837 CallDescrWorkerInternal + 124
106 libcoreclr.dylib              	0x00000001040adc04 MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 964
107 libcoreclr.dylib              	0x00000001040c9295 QueueUserWorkItemManagedCallback(void*) + 165
108 libcoreclr.dylib              	0x0000000104070ba0 ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 416
109 libcoreclr.dylib              	0x0000000104071303 ManagedThreadBase::ThreadPool(ADID, void (*)(void*), void*) + 51
110 libcoreclr.dylib              	0x00000001040654ac ManagedPerAppDomainTPCount::DispatchWorkItem(bool*, bool*) + 268
111 libcoreclr.dylib              	0x0000000104090eef ThreadpoolMgr::WorkerThreadStart(void*) + 1103
112 libcoreclr.dylib              	0x0000000103ee5648 CorUnix::CPalThread::ThreadEntry(void*) + 328
113 libsystem_pthread.dylib       	0x00007fff5997f2eb _pthread_body + 126
114 libsystem_pthread.dylib       	0x00007fff59982249 _pthread_start + 66
115 libsystem_pthread.dylib       	0x00007fff5997e40d thread_start + 13

Comment by Richard Collette [ 31/Aug/19 ]

I just noticed that this may be related to CSHARP-2308

Generated at Wed Feb 07 21:43:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.