[CSHARP-1705] IQueryable - Join operation fails following previous filter and join Created: 12/Jul/16  Updated: 31/Aug/18  Resolved: 31/Aug/18

Status: Closed
Project: C# Driver
Component/s: Linq
Affects Version/s: 2.2.4
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Andrew Bennett Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 7


Attachments: Zip Archive JoinReproduction.zip    

 Description   

When using the Join operation on an IQueryable the operation throws an exception in the following circumstances

1. Join one typed collection's items to another on an integer field; Queryable of Item A to Queryable Item B
2. Filter the resultant items (Item B) using a where for example
3. Join the the resultant items back to their parent items; Queryable of Item B to Queryable of Item A
4. Execute the resultant query via a call to toArray(), giving an array of item A.

On the call to toArray() the following exception is thrown:

Test method MongoInvestigation.Test.UnitTest1.TestMethod1 threw exception: 
System.ArgumentException: Expression of type 'System.Collections.Generic.IEnumerable`1[MongoInvestigation.Test.UnitTest1+Order]' cannot be used for parameter of type 'System.Linq.IQueryable`1[MongoInvestigation.Test.UnitTest1+Order]' of method 'System.Linq.IQueryable`1[MongoInvestigation.Test.UnitTest1+Product] Join[Order,Product,Int32,Product](System.Linq.IQueryable`1[MongoInvestigation.Test.UnitTest1+Order], System.Collections.Generic.IEnumerable`1[MongoInvestigation.Test.UnitTest1+Product], System.Linq.Expressions.Expression`1[System.Func`2[MongoInvestigation.Test.UnitTest1+Order,System.Int32]], System.Linq.Expressions.Expression`1[System.Func`2[MongoInvestigation.Test.UnitTest1+Product,System.Int32]], System.Linq.Expressions.Expression`1[System.Func`3[MongoInvestigation.Test.UnitTest1+Order,MongoInvestigation.Test.UnitTest1+Product,MongoInvestigation.Test.UnitTest1+Product]])'
    at System.Linq.Expressions.Expression.ValidateOneArgument(MethodBase method, ExpressionType nodeKind, Expression arg, ParameterInfo pi)
   at System.Linq.Expressions.Expression.ValidateArgumentTypes(MethodBase method, ExpressionType nodeKind, ReadOnlyCollection`1& arguments)
   at System.Linq.Expressions.Expression.Call(Expression instance, MethodInfo method, IEnumerable`1 arguments)
   at System.Linq.Expressions.MethodCallExpressionN.Rewrite(Expression instance, IList`1 args)
   at MongoDB.Driver.Linq.Processors.SerializationBinder.Visit(Expression node)
   at System.Linq.Expressions.ExpressionVisitor.Visit(ReadOnlyCollection`1 nodes)
   at MongoDB.Driver.Linq.Processors.SerializationBinder.BindEmbeddedPipeline(MethodCallExpression node)
   at MongoDB.Driver.Linq.Processors.SerializationBinder.Visit(Expression node)
   at MongoDB.Driver.Linq.Processors.Pipeline.MethodCallBinders.JoinBinder.Bind(PipelineExpression pipeline, PipelineBindingContext bindingContext, MethodCallExpression node, IEnumerable`1 arguments)
   at MongoDB.Driver.Linq.Processors.MethodInfoMethodCallBinder`1.Bind(PipelineExpression pipeline, TBindingContext bindingContext, MethodCallExpression node, IEnumerable`1 arguments)
   at MongoDB.Driver.Linq.Processors.PipelineBinderBase`1.BindMethodCall(MethodCallExpression node)
   at MongoDB.Driver.Linq.Processors.PipelineBinderBase`1.Bind(Expression node)
   at MongoDB.Driver.Linq.Processors.Pipeline.PipelineBinder.Bind(Expression node, IBsonSerializerRegistry serializerRegistry)
   at MongoDB.Driver.Linq.MongoQueryProviderImpl`1.Prepare(Expression expression)
   at MongoDB.Driver.Linq.MongoQueryProviderImpl`1.Translate(Expression expression)
   at MongoDB.Driver.Linq.MongoQueryProviderImpl`1.Execute(Expression expression)
   at MongoDB.Driver.Linq.MongoQueryableImpl`2.GetEnumerator()
   at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
   at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
   at MongoInvestigation.Test.UnitTest1.<TestMethod1>d__2.MoveNext() in C:\GITProjects\MongoInvestigation\MongoInvestigation.Test\UnitTest1.cs:line 46
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()

Please see attached code for full reproduction



 Comments   
Comment by Craig Wilson [ 19/Aug/16 ]

Hi Andrew,

Sorry about the wait. I missed your comment. So, I chased down the wrong issue with your query. Your answer is much simpler. I've changed your code to the following:

/*Act*/
            //Join Products to Orders
            var productsForOrders = ordersCollection.AsQueryable()
                .Join(
                    productsCollection.AsQueryable(),
                    order => order.ProductId,
                    product => product.Id,
                    (order, product) => product);
 
            //Filter products
            var productsMatching = productsForOrders.Where(x => x.Name == "Fries");
 
            //Join resultant products back to orders
            var ordersJoinProducts = productsMatching // ***NOTE: This is in a different order. We need to join from the pipeline to a collection, not a collection to a pipeline.
                .Join(
                    ordersCollection.AsQueryable(),
                    product => product.Id,
                    order => order.ProductId,
                    (product, order) => order);
 
            var agg = ordersJoinProducts.ToString();
 
            //execute query
            var results = ordersJoinProducts.ToArray();
 
            /*Assert*/
            Assert.AreEqual(results.Length, 1);
            Assert.AreEqual(1, results.Single().Id);

As commented in the code, MongoDB's $lookup stage can't join to another pipeline, only to another collection. So, in this case, (and because this is an inner join), we can reverse the inner and outer entities. This generates the below pipeline, for those interested:

db.orders.aggregate([
 
{ "$lookup" : { "from" : "products", "localField" : "ProductId", "foreignField" : "_id", "as" : "product" } }, 
{ "$unwind" : "$product" }, 
{ "$project" : { "product" : "$product", "_id" : 0 } }, 
{ "$match" : { "product.Name" : "Fries" } }, 
{ "$lookup" : { "from" : "orders", "localField" : "product._id", "foreignField" : "ProductId", "as" : "order" } }, 
{ "$unwind" : "$order" }, 
{ "$project" : { "order" : "$order", "_id" : 0 } }
 
])

The title of this ticket is actually correct. We cannot join to something that has a previous anything, be it a filter, join, select, etc... This is a MongoDB limitation and there is nothing we can do about it. Just to note, MongoDB 3.4 will be supporting readonly views, which are defined as a pipeline. From .NET, you'd use them like a collection. This might allow you to get around the $lookup restriction.

Hope that helps, and my apologies for mis-diagnosing this initially.
Craig

Comment by Andrew Bennett [ 05/Aug/16 ]

Hi Craig,

Thanks for looking into this for us. Any chance you could modify my reproduction to demonstrate this working? I think we tried this ourselves and had issues but it would be really clear if you could upload our reproduction corrected to project anonymous types.

Thanks
Andrew

Comment by Craig Wilson [ 05/Aug/16 ]

Hi Andrew,

So, I'm able to do n-ary joins without an issue. I believe the ultimate problem with your example is that you are projecting entire entities. For instance, you join Product to Order and then your result select is the Product. Unfortunately, there is no good way to handle this in MongoDB 3.2. For now, you'll need to project into an anonymous type or a new entity or something. For instance, simply projecting new

{product, order}

would be sufficient.

In MongoDB 3.4, this will be possible by using an exclusion projection. However, until then, we can't really do anything substantial. I'm going to move this ticket to driver release 2.4 with a dependency on the 3.4 server.

Craig

Comment by Andrew Bennett [ 15/Jul/16 ]

Hi any idea when this issue will be looked into? This and C-SHARP-1702 are critical to a bug we need to fix. Thanks

Generated at Wed Feb 07 21:40:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.