[CSHARP-4548] IFindFluent.Projection fails with ExpressionNotSupportedException after upgrading to 2.19.0 Created: 25/Feb/23  Updated: 27/Oct/23  Resolved: 22/Mar/23

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: GDar N/A Assignee: Robert Stam
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on CSHARP-4549 Support Tuple.Create and ValueTuple.C... Closed
Duplicate
duplicates CSHARP-4498 .Project(c => c.Reference()) not work Closed
duplicates CSHARP-4763 Consider supporting client side proje... Scheduled
Documentation Changes Summary:

1. What would you like to communicate to the user about this feature?
2. Would you like the user to see examples of the syntax and/or executable code and its output?
3. Which versions of the driver/connector does this apply to?


 Description   

Summary

I've always relied on the driver's projection functionality using LINQ expressions. Before version 2.19.0, it worked well, selecting only those fields, that were used in the projection expression, translating known functions into the mongo functions, and doing client-side projections for the functions that were unable to be translated.

But now, in 2.19.0, I can no longer use functions that I want to run client-side. Let's say I have a BsonDocument collection and want to select only 2 fields, then I want to do a client-side projection using those 2 fields and my custom function. I used to just write:

var find =
    mongoCollection
        .Find(document => true)
        .Project(document => Functions.MyClientSideProjectionFunction(document["_key"], document["prop"])); 

and it used to work well, but now, it fails with this error: 

MongoDB.Driver.Linq.ExpressionNotSupportedException: Expression not supported: MyClientSideProjectionFunction(document.get_Item("_key"), document.get_Item("prop")). 

You can't even make a tuple with those selected properties to map them later. ValueTuple.Create and Tuple.Create functions also fail.

This happens at the library level, so it doesn't matter which server configuration you use.



 Comments   
Comment by Robert Stam [ 22/Mar/23 ]

`IAsyncEnumerable` isn't implemented because it hasn't always existed and still doesn't exist in all our targeted frameworks. We definitely plan to implement it some day.

You don't have to use `ToList` to use client side projections. The examples above separated the "find" part from the "client side projection" part but they could be written as a single chain of calls, none of which actually call the database until the final enumerable is enumerated:

var find = collection
    .Find(_ => true)
    .Project(x => new { Key = x.Key, Prop = x.Prop });
    .ToEnumerable()
    .Select(x => MyClientSideProjection(x.Key, x.Prop)); // this will execute client side but not until "find" is enumerated

The role of calling `ToEnumerable()` here is to separate the database operation (the `IQueryable` part) from the client side projection (the `IEnumerable` part). None of the code above calls the database. The database would be called when enumerating find, for example by executing:

var results = find.ToList();

but of course that requires enough memory to hold the entire result in memory at once.

You could also iterate through the results using:

foreach (var result in find)
{
    // process result
}

It's a little harder to incorporate client side projection with async processing. You would have to factor out the client side processing to be separate from fetching the database results. It would look something like the following:

var find = collection
    .Find(_ => true)
    .Project(x => new { Key = x.Key, Prop = x.Prop });
 
var cursor = await find.ToCursorAsync();
while (await cursor.MoveNextAsync())
{
    foreach (var document in cursor.Current)
    {
        var result = MyClientSideProjection(document.Key, document.Prop);
        // process result
    }
}

Comment by GDar N/A [ 17/Mar/23 ]

Thank you for the clarification. So, as this change is intentional, can we get the ability to do a client-side projection with minimal overhead? For now, all the ways to do it is to, either use a non-async IEnumerable .ToList which will block the thread, or use ToListAsync which will allocate extra memory. Of course, there is a way to traverse a Cursor (which strangely doesn't implement the IAsyncEnumerable interface, but it's another topic.) and accumulate projected documents in your list, but it's still a step back in terms of optimizations, because when it's done inside the driver, it has a better context of the data, like it's total count, etc to do it in the best way possible.

Comment by Robert Stam [ 27/Feb/23 ]

Thank you for reporting this issue. As explained in CSHARP-4498 we no longer support client side projections. If you want to do a client side projection you should do so explicitly. We don't want to hide the fact that data is being brought from the server and then being processed client side.

There are several techniques for a temporary data object to contain the data being retrieved from the server: BsonDocument, anonymous class, explicit C# class, Tuple and ValueTupe would be the most common.

BsonDocument, anonymous classes and explicit C# classes work right now.

Tuple works already if you call the constructor instead of the `Tuple.Create` method. I've created CSHARP-4549 to support calling the `Tuple.Create` method.

ValueTuple will be supported either calling the constructor or calling `ValueTuple.Create` once CSHARP-4490 and CSHARP-4549 are done.

I'm writing test cases to demonstrate how all the above approaches would work.

Here's the first one that you could use right now without waiting for a new release:

var find = collection
    .Find(_ => true)
    .Project(Builders<BsonDocument>.Projection.Include("key").Include("prop").Exclude("_id"));
 
var results = find
    .ToEnumerable()
    .Select(x => MyClientSideProjection(x["key"], x["prop"])) // this executes client side
    .ToList();

Here's a similar example using Tuple (calling the constructor instead of the Create method):

var find = collection
    .Find(_ => true)
    .Project(x => new Tuple<BsonValue, BsonValue>(x["key"], x["prop"]));
 
var results = find
    .ToEnumerable()
    .Select(x => MyClientSideProjection(x.Item1, x.Item2)) // this executes client side    
    .ToList();

Generated at Wed Feb 07 21:48:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.