[CSHARP-4054] One-to-many LINQ query using Join fails with InvalidOperationException Created: 11/Feb/22  Updated: 21/Apr/23

Status: Backlog
Project: C# Driver
Component/s: LINQ3
Affects Version/s: 2.14.1
Fix Version/s: None

Type: Bug Priority: Unknown
Reporter: James Kovacs Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: triage
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to CSHARP-4053 One-to-many LINQ query fails with una... Backlog
Documentation Changes: Not Needed
Documentation Changes Summary:

1. What would you like to communicate to the user about this feature?
2. Would you like the user to see examples of the syntax and/or executable code and its output?
3. Which versions of the driver/connector does this apply to?


 Description   

If you try to reformulate the query in CSHARP-4053 using a join to the person.MovieIds enumerable, an InvalidOperationException is thrown:

using System;
using System.Collections.Generic;
using System.Linq;
using MongoDB.Bson;
using MongoDB.Driver;
using MongoDB.Driver.Linq;
 
var settings = new MongoClientSettings { LinqProvider = LinqProvider.V3 };
var client = new MongoClient(settings);
var db = client.GetDatabase("test");
var people = db.GetCollection<Person>("people");
var movies = db.GetCollection<Movie>("movies");
 
var query = from person in people.AsQueryable()
            from movieId in person.MovieIds
            join movie in movies.AsQueryable() on movieId equals movie.Id
            select new { person, movie };
 
foreach (var tuple in query.ToList())
{
    Console.WriteLine(tuple);
}
 
class Movie
{
    public ObjectId Id { get; set; }
}
 
class Person
{
    public IEnumerable<ObjectId> MovieIds { get; set; }
}

The exception is:

/Users/james/Dropbox/code/mongodb/mongo-csharp-driver/tests/MongoDB.Driver.TestConsoleApplication/bin/Debug/netcoreapp3.1/MongoDB.Driver.TestConsoleApplication 
Unhandled exception. System.InvalidOperationException: Operation is not valid due to the current state of the object.
   at MongoDB.Driver.Linq.Linq3Implementation.Serializers.WrappedValueSerializer`1.TryGetMemberSerializationInfo(String memberName, BsonSerializationInfo& serializationInfo) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/Serializers/WrappedValueSerializer.cs:line 77
   at MongoDB.Driver.Linq.Linq3Implementation.Misc.DocumentSerializerHelper.HasFieldInfo(IBsonSerializer serializer, String memberName) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/Misc/DocumentSerializerHelper.cs:line 40
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToAggregationExpressionTranslators.MemberExpressionToAggregationExpressionTranslator.Translate(TranslationContext context, MemberExpression expression) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToAggregationExpressionTranslators/MemberExpressionToAggregationExpressionTranslator.cs:line 46
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToAggregationExpressionTranslators.ExpressionToAggregationExpressionTranslator.Translate(TranslationContext context, Expression expression) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToAggregationExpressionTranslators/ExpressionToAggregationExpressionTranslator.cs:line 67
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToAggregationExpressionTranslators.ExpressionToAggregationExpressionTranslator.TranslateLambdaBody(TranslationContext context, LambdaExpression lambdaExpression, IBsonSerializer parameterSerializer, Boolean asRoot) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToAggregationExpressionTranslators/ExpressionToAggregationExpressionTranslator.cs:line 115
   at MongoDB.Driver.Linq.Linq3Implementation.Misc.LambdaExpressionExtensions.GetFieldPath(LambdaExpression fieldSelectorLambda, TranslationContext context, IBsonSerializer parameterSerializer) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/Misc/LambdaExpressionExtensions.cs:line 27
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.JoinMethodToPipelineTranslator.Translate(TranslationContext context, MethodCallExpression expression) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToPipelineTranslators/JoinMethodToPipelineTranslator.cs:line 52
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.ExpressionToPipelineTranslator.Translate(TranslationContext context, Expression expression) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToPipelineTranslators/ExpressionToPipelineTranslator.cs:line 44
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToExecutableQueryTranslators.ExpressionToExecutableQueryTranslator.Translate[TDocument,TOutput](MongoQueryProvider`1 provider, Expression expression) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/Translators/ExpressionToExecutableQueryTranslators/ExpressionToExecutableQueryTranslator.cs:line 34
   at MongoDB.Driver.Linq.Linq3Implementation.MongoQuery`2.Execute() in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/MongoQuery.cs:line 66
   at MongoDB.Driver.Linq.Linq3Implementation.MongoQuery`2.GetEnumerator() in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/src/MongoDB.Driver/Linq/Linq3Implementation/MongoQuery.cs:line 78
   at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Program.<Main>$(String[] args) in /Users/james/Dropbox/code/mongodb/mongo-csharp-driver/tests/MongoDB.Driver.TestConsoleApplication/Program.cs:line 19



 Comments   
Comment by Alistair Steele [ 13/Mar/23 ]

The following shows the issue, looks to happen when using a SelectMany with a resultSelector.

using MongoDB.Driver;
using MongoDB.Driver.Linq;
using static System.Linq.Queryable;
 
public class Program
{
    public class Info
    {
        public Guid Id;
        public string[] FooNames;
    }    
 
    public class Foo
    {
        public Guid Id;
        public string FooName;
    }    
 
    public static void Main(string[] args)
    {
        // Setup 
        string uri = "mongodb://localhost:27017";
        MongoClientSettings settings = MongoClientSettings.FromConnectionString(uri);
        settings.LinqProvider = LinqProvider.V3;
        MongoClient client = new(settings: settings);
        IMongoDatabase database = client.GetDatabase("db");
        IMongoCollection<Info> infos = database.GetCollection<Info>("Info");
        IMongoCollection<Foo> foos = database.GetCollection<Foo>("Foo");
        infos.DeleteMany(x => true);
        foos.DeleteMany(x => true);        
 
        // Test data
        infos.InsertOne(new() { FooNames = new string[] { "foo1", "foo2" } });
        infos.InsertOne(new() { FooNames = new string[] { "foo2" } });
        infos.InsertOne(new() { FooNames = new string[] { "foo2", "foo3" } });
        foos.InsertOne(new() { FooName = "foo1" });
        foos.InsertOne(new() { FooName = "foo2" });
        foos.InsertOne(new() { FooName = "foo3" });        
 
        // Queryables
        IQueryable<Info> infosQ = infos.AsQueryable();
        IQueryable<Foo> foosQ = foos.AsQueryable();        
 
        // Flatten Infos
        var flatInfos = infosQ.SelectMany(x => x.FooNames, (x, y) => new
        {
            Id = x.Id,
            Name = y
        });        
 
        // Join each flattened Info the matching Foo
        var infoAndFoo = flatInfos.GroupJoin(foosQ, x => x.Name, y => y.FooName, (x, y) => new
        {
            InfoId = x.Id,
            Foo = y
        });        
 
        // Outputs but the query formats to a captured exception (below)
        Console.WriteLine($"Query = {infoAndFoo}");
    }
} 

Results in the output:

Query = System.InvalidOperationException: Operation is not valid due to the current state of the object.
   at MongoDB.Driver.Linq.Linq3Implementation.Serializers.WrappedValueSerializer`1.TryGetMemberSerializationInfo(String memberName, BsonSerializationInfo& serializationInfo)
   at MongoDB.Driver.Linq.Linq3Implementation.Misc.DocumentSerializerHelper.HasMemberSerializationInfo(IBsonSerializer serializer, String memberName)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToAggregationExpressionTranslators.MemberExpressionToAggregationExpressionTranslator.Translate(TranslationContext context, MemberExpression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToAggregationExpressionTranslators.ExpressionToAggregationExpressionTranslator.Translate(TranslationContext context, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Misc.LambdaExpressionExtensions.GetFieldPath(LambdaExpression fieldSelectorLambda, TranslationContext context, IBsonSerializer parameterSerializer)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.GroupJoinMethodToPipelineTranslator.Translate(TranslationContext context, MethodCallExpression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.ExpressionToPipelineTranslator.Translate(TranslationContext context, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToExecutableQueryTranslators.ExpressionToExecutableQueryTranslator.Translate[TDocument,TOutput](MongoQueryProvider`1 provider, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.MongoQuery`2.ToString()

Applying the same fix you did for .Join fixes the issue in GroupJoin (tried locally)

Comment by Robert Stam [ 13/Mar/23 ]

Thanks for the additional information. Do you have any sample code I could use to reproduce this?

Comment by Alistair Steele [ 10/Mar/23 ]

I'm getting a very similar stack trace (throwing InvalidOperationException in WrappedValueSerializer) with GroupJoin, after looking at the fix made it seems that fix may need to be applied to GroupJoin as well? @rstam

Comment by Githook User [ 22/Feb/23 ]

Author:

{'name': 'rstam', 'email': 'robert@robertstam.org', 'username': 'rstam'}

Message: CSHARP-4054: One-to-many LINQ query using Join fails with InvalidOperationException.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/1eaff4dbf187b41e4a1733471f9b87d20135ff47

Comment by Robert Stam [ 15/Feb/23 ]

Thanks for bringing this to our attention. I will look into it.

Comment by Matt Allwood [ 15/Feb/23 ]

Still an issue with 2.19, which is an issue now that v3 is the default implementation used. Still working with v2 LINQ driver

Similar stack trace to above, triggered by WrappedValueSerializer: 

  at MongoDB.Driver.Linq.Linq3Implementation.Serializers.WrappedValueSerializer`1.TryGetMemberSerializationInfo(String memberName, BsonSerializationInfo& serializationInfo)
   at MongoDB.Driver.Linq.Linq3Implementation.Misc.DocumentSerializerHelper.HasMemberSerializationInfo(IBsonSerializer serializer, String memberName)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToAggregationExpressionTranslators.MemberExpressionToAggregationExpressionTranslator.Translate(TranslationContext context, MemberExpression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToAggregationExpressionTranslators.ExpressionToAggregationExpressionTranslator.Translate(TranslationContext context, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Misc.LambdaExpressionExtensions.GetFieldPath(LambdaExpression fieldSelectorLambda, TranslationContext context, IBsonSerializer parameterSerializer)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.JoinMethodToPipelineTranslator.Translate(TranslationContext context, MethodCallExpression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.ExpressionToPipelineTranslator.Translate(TranslationContext context, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.WhereMethodToPipelineTranslator.Translate(TranslationContext context, MethodCallExpression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.ExpressionToPipelineTranslator.Translate(TranslationContext context, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.OrderByMethodToPipelineTranslator.Translate(TranslationContext context, MethodCallExpression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.ExpressionToPipelineTranslator.Translate(TranslationContext context, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.GroupByMethodToPipelineTranslator.Translate(TranslationContext context, MethodCallExpression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.ExpressionToPipelineTranslator.Translate(TranslationContext context, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.SelectMethodToPipelineTranslator.Translate(TranslationContext context, MethodCallExpression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToPipelineTranslators.ExpressionToPipelineTranslator.Translate(TranslationContext context, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.Translators.ExpressionToExecutableQueryTranslators.ExpressionToExecutableQueryTranslator.Translate[TDocument,TOutput](MongoQueryProvider`1 provider, Expression expression)
   at MongoDB.Driver.Linq.Linq3Implementation.MongoQuery`2.ToCursorAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken) 

Comment by Joris Laperre [ 26/Feb/22 ]

Hi,

Fyi, I came across this too, presumably it's the same root cause, code below. The error is not happening in Linq V2 btw (so I'm going to use that as a workaround for now).

using System;
using System.Collections.Generic;
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Driver;
using MongoDB.Driver.Linq;namespace BugReport2
{
    [BsonIgnoreExtraElements]
    public class OrderItem
    {
        [BsonElement("orderQty"), BsonRequired]
        public int OrderQty { get; set; }
        [BsonElement("productID"), BsonRequired]
        public int ProductID { get; set; }
    }
 
    [BsonIgnoreExtraElements]
    public class Order
    {
        [BsonId]
        public int _id { get; set; }
        [BsonElement("orderItems")]
        public List<OrderItem> OrderItems { get; set; }
    }
 
    [BsonIgnoreExtraElements]
    public class Product
    {
        [BsonId]
        public int _id { get; set; }
        [BsonElement("name"), BsonRequired]
        public string Name { get; set; }
    }
 
    class Program
    {
        static void Main(string[] args)
        {
            var connectionString = "mongodb://localhost";
            var clientSettings = MongoClientSettings.FromConnectionString(connectionString);
            clientSettings.LinqProvider = LinqProvider.V3;
            var mongoClient = new MongoClient(clientSettings);
            var db = mongoClient.GetDatabase("test");
            var productsCollection = db.GetCollection<Product>("products");
            var ordersCollection = db.GetCollection<Order>("orders");
            if (productsCollection.Find(p => p._id == 1).CountDocuments() == 0)
            {
                productsCollection.InsertOne(new Product { _id = 1, Name = "New Product" });
            }
            if (ordersCollection.Find(o => o._id == 1).CountDocuments() == 0)
            {
                ordersCollection.InsertOne(new Order { _id = 1, OrderItems = new List<OrderItem>() { new OrderItem { ProductID = 1, OrderQty = 1 } } });
            }
            var x = ordersCollection.AsQueryable()
                .Where(o => o._id == 1)
                .SelectMany(o => o.OrderItems, (k, o) => new { OrderID = k._id, Quantity = o.OrderQty, ProductID = o.ProductID })
                .Join(productsCollection.AsQueryable(), o => o.ProductID, p => p._id, (x, y) => new { OrderID = x.OrderID, Quantity = x.Quantity, Name = y.Name })
                .ToList()
                ;
        }
    }
}

Generated at Wed Feb 07 21:47:04 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.