[CSHARP-1555] Projecting from and to the same type including the _id field causes deserialization to fail Created: 03/Feb/16  Updated: 04/Jul/22  Resolved: 04/Jul/22

Status: Closed
Project: C# Driver
Component/s: Linq, LINQ3
Affects Version/s: 2.2
Fix Version/s: 2.17.0

Type: Bug Priority: Minor - P4
Reporter: Roberto Pérez Assignee: Robert Stam
Resolution: Done Votes: 1
Labels: triaged
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by CSHARP-1606 Error using Select function Closed
Related
is related to CSHARP-1719 Selecting from a IMongoQueryable coll... Closed
is related to CSHARP-1745 Deserialization issue when using GroupBy Closed
Epic Link: CSHARP-3615
Backwards Compatibility: Fully Compatible

 Description   

When using: yourMongoCollection.AsQueryable().Select(projectionExpression).ToList() in versions greater than 2.1.X it fails.

The examples below just call Count() for demonstration purposes.

Program.cs

    internal class Program
    {
        internal class Person
        {
            public Guid Id { get; set; }
            public string Name { get; set; }
        }
 
        private static void Main(string[] args)
        {
            // RegisterConventions
            var conventionPack = new ConventionPack
            {
                new IgnoreIfNullConvention(true),
                new NoIdMemberConvention()
            };
            ConventionRegistry.Register("GlobalConventions", conventionPack, t => true);
 
            // Mapping People collection
            var map = BsonClassMap.RegisterClassMap<Person>();
            map.AutoMap();
            map.MapProperty(c => c.Id).SetIsRequired(true);
            map.SetIdMember(map.GetMemberMap(c => c.Id));
 
            // Creating mongo client and dropping collection
            var mongoClient = new MongoClient("mongodb://buildbee");
            var mongoDatabase = mongoClient.GetDatabase("db");
            mongoDatabase.DropCollectionAsync("people").Wait();
            var peopleCollection = mongoDatabase.GetCollection<Person>("people");
            
            // one document is inserted
            peopleCollection.InsertOneAsync(new Person
            {
                Id = Guid.NewGuid(),
                Name = "A"
            }).Wait();
 
          
            DoIt("1- Count", () => peopleCollection.AsQueryable().Count());
 
            DoIt("2- ToList Without Select", () => peopleCollection.AsQueryable().ToList().Count);
 
            DoIt("3.1- ToList With Select", () => peopleCollection.AsQueryable().Select(p => p).ToList().Count);
 
            DoIt("3.2- ToList With Select", () => peopleCollection.AsQueryable().Select(p => new Person
            {
                Id = p.Id,
                Name = p.Name,
            }).ToList().Count);
 
            DoIt("3.3- ToList With Select", () => peopleCollection.AsQueryable().Select(p => new Person
            {
                Id = p.Id,
                // Name = p.Name,
            }).ToList().Count);
 
            DoIt("3.4- ToList With Select", () => peopleCollection.AsQueryable().Select(p => new Person
            {
                // Id = p.Id,
                Name = p.Name,
            }).ToList().Count);
        }
 
        private static void DoIt(string message, Func<int> countQuery)
        {
            try
            {
                Console.WriteLine(message);
                Console.WriteLine(countQuery());
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
            Console.WriteLine();
        }
    }

Execution output using .NEt Driver 2.1.1 or 2.1.0

1- Count
1

2- ToList Without Select
1

3.1- ToList With Select
1

3.2- ToList With Select
1

3.3- ToList With Select
1

3.4- ToList With Select
1

Press any key to continue . . .

Execution output using .NEt Driver 2.2.0 or greater

1- Count
1

2- ToList Without Select
1

3.1- ToList With Select
1

3.2- ToList With Select
System.FormatException: Element 'Id' does not match any field or property of class ConsoleApplication1.Program+Person.
at MongoDB.Driver.Linq.MongoQueryProviderImpl`1.Execute(Expression expression)
at MongoDB.Driver.Linq.MongoQueryableImpl`2.GetEnumerator()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at ...

3.3- ToList With Select
System.FormatException: Element 'Id' does not match any field or property of class ConsoleApplication1.Program+Person.
at MongoDB.Driver.Linq.MongoQueryProviderImpl`1.Execute(Expression expression)
at MongoDB.Driver.Linq.MongoQueryableImpl`2.GetEnumerator()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at ...

3.4- ToList With Select
System.FormatException: Required element '_id' for property 'Id' of class ConsoleApplication1.Program+Person is missing.
at MongoDB.Driver.Linq.MongoQueryProviderImpl`1.Execute(Expression expression)
at MongoDB.Driver.Linq.MongoQueryableImpl`2.GetEnumerator()
at System.Collections.Generic.List`1..ctor(IEnumerable`1 collection)
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
at ...

Press any key to continue . . .



 Comments   
Comment by James Kovacs [ 04/Jul/22 ]

This issue has been fixed in the new LINQ provider (known as LINQ3), which was introduced in the 2.14 release.

Configure your MongoClientSettings to use LinqProvider.V3 if you want to use this functionality.

To configure a client to use the LINQ3 provider use code like the following

var connectionString = "mongodb://localhost";
var clientSettings = MongoClientSettings.FromConnectionString(connectionString);
clientSettings.LinqProvider = LinqProvider.V3;
var client = new MongoClient(clientSettings);

Comment by Githook User [ 04/Jul/22 ]

Author:

{'name': 'Robert Stam', 'email': 'robert@robertstam.org', 'username': 'rstam'}

Message: CSHARP-1555: Verify that issue is not present in LINQ3. (#838)
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/261066009ac6b315a2c215791e27a3ac4e54735f

Comment by Roberto Pérez [ 11/May/16 ]

Hello Craig,

After new attempts to workaround the problem we have found out more scenarios.

Projecting from and to the same type including the _id field causes deserialization to fail: this is only true if you have not mark a property as Required

map.MapProperty(c => c.Id).SetIsRequired(true);

in any other case no matter which fields you select for your projection, it will not work if the same type is used for projection

Regards

Comment by Craig Wilson [ 04/Apr/16 ]

Understood. I'll keep thinking and hopefully come up with a better solution. This is just a very specific set of circumstances that I don't think many people will encounter.

Comment by Roberto Pérez [ 04/Apr/16 ]

Hi Craig,

First of all thank you very much for your work.

In the examples shown here we noticed that anonymous classes elude the mentioned problem but as we were using the same type in projections with the Identifier in previous versions of the Driver we wouldn't like to change those pieces of code (the application is huge and we have built extension methods using generics to specify the projection in a layer on top of the Driver; so anonymous classes would not work for us and we would not like to create as many View classes as we would need in each scenario).

We are still using .NET Driver version 2.1.1, anyway, if you come up with anything else, please tell us

Comment by Craig Wilson [ 04/Apr/16 ]

One further note - This problem only manifest in the following conditions:

1) You are projecting to the same type. In the description example, both the original and the target type are the same.
2) You are both projecting the identifier (_id) as well as set it's required status to true.

After further looking, this is a non-trivial fix. When you need to project an identifier, I'd suggest you use either anonymous types or create a new XXXView class to project into.

Comment by Craig Wilson [ 04/Apr/16 ]

I have this down to a simple test we can add to MongoQueryableTests.cs.

[Test]
        public void Select_named_type_syntax()
        {
            var query = from x in CreateQuery()
                        select new Root { Id = x.Id, A = x.A };
 
            Assert(query,
                2,
                "{ $project: { Id: '$_id', A: '$A', _id: 0 } }");
        }

I believe this issue is a couple of things rolled into one related to when to exclude projecting the _id field. We didn't have a test around this, so when some stuff changed, we didn't catch this change. Thanks for reporting.

Comment by Roberto Pérez [ 04/Feb/16 ]

Hi Craig,

Answering your questions:
1) We are setting new NoIdMemberConvention() just because we had problems with embedded documents and properties named Id in those documents. (I am not sure at 100 percent but I remember so).

2) The example of person was just to reproduce the error (this case is so simple that it is nonsense).
But imagine that we have a class Person with two complex properties and depending on something we want to retrieve:
a) the whole document
b) the document without one of these complex properties
c) the document without the other complex property
d) the document without none of these complex properties
We do not want to handle anonymous classes and we do not want to create different objects just for these projections either, so we reuse the main class Person but knowing that in same cases some properties are going to be null.

I have comment the NoIdMemberConvention line but the result is the same:

            // RegisterConventions
            var conventionPack = new ConventionPack
            {
                new IgnoreIfNullConvention(true),
          //      new NoIdMemberConvention()
            };
            ConventionRegistry.Register("GlobalConventions", conventionPack, t => true);

Regards

Comment by Craig Wilson [ 04/Feb/16 ]

In your examples, Person2 and the anonymous type also use a BsonClassMap, even though that part is hidden from you. In fact, in projections, we ignore any existing class maps for a number of reasons, but mostly due to compability with the instigating type. This is a guess, but since you have registered a NoIdMemberConvention, I imagine that this could be getting confused and not mapping the identifier properly. I'm not sure why the other forms work, so could you run a test and remove that from the conventions.

I have 2 questions:
1) Why are you globally setting a no id convention?
2) Why, if you already have a Person (in the p lambda parameter), are you constructing a new one of the same type?

Craig

Comment by Roberto Pérez [ 04/Feb/16 ]

Hi Craig,

The projection forces the error only when the class type used is one mapped in BsonClassMap.
Therefore, a projection with an anonymous class works and a projection with another class also works

        internal class Person2
        {
            public Guid Id { get; set; }
            public string Name { get; set; }
        }
 
.....
            DoIt("Works", () => peopleCollection.AsQueryable().Select(p => new Person2
            {
                Id = p.Id,
                Name = p.Name,
            }).ToList().Count);

        DoIt("Works", () => peopleCollection.AsQueryable().Select(p => new 
            {
                Id = p.Id,
                Name = p.Name,
            }).ToList().Count);

            DoIt("Does not work", () => peopleCollection.AsQueryable().Select(p => new Person
            {
                Id = p.Id,
                Name = p.Name,
            }).ToList().Count);

Regards

Comment by Craig Wilson [ 03/Feb/16 ]

Hi Roberto,

I'll take a look and try and repro. This seems pretty straightforward, so I'm surprised it doesn't already work given that we have lots of users currently doing things exactly like this. Perhaps we are just missing a test?

Craig

Generated at Wed Feb 07 21:39:58 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.