[CSHARP-4618] Cannot find documentation on differences between LINQ2 and LINQ3 providers Created: 17/Apr/23  Updated: 26/Jan/24

Status: Scheduled
Project: C# Driver
Component/s: LINQ3
Affects Version/s: 2.19.1
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: Thomas Tutko Assignee: Robert Stam
Resolution: Unresolved Votes: 4
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to CSHARP-4079 Implement positional update operators... Closed

 Description   

Summary

The latest MongoDB.Driver for C# switched the default LINQ Provider from 2 to 3 but is severely lacking in the "Breaking Changes" section of the release notes. We updated our project to use the latest driver, addressed the issues called out in breaking changes such as the ObjectSerializer changes.

We then began seeing errors like the following:
"ExpressionNotSupportedException: ... because negative indexes are not valid. To use the positional operator $ use FirstMatchingElement instead of an index value of -1."

 

Great.. so clearly based on the error message, this is a "known" change but I had to do some web searching just to find https://jira.mongodb.org/browse/CSHARP-4079 which nonchalantly says something like "Oh, we're going to be replacing these 3 things with these 3 extension methods" and closes the issue. That's all well and good except where is this called out in the documentation or more importantly the breaking changes section? Where is the page that documents what to look out for or change when moving from LINQ2 to LINQ3 provider? If it exists, that's great but it also needs to be more discoverable and should be listed in the breaking changes section of the 2.19.0 driver release. If it doesn't exist, please create one because I have no idea what else is going to bite me in this code base due to undocumented changes (at least undocumented from the standpoint of I couldn't find anything).

 

I will try the MongoDB Analyzer now as I was not previously aware of this and it is not called out in the release notes for 2.19.x.

 



 Comments   
Comment by Sys Ops [ 19/Oct/23 ]

robert@mongodb.com  Another thing that would greatly help is when all LINQ3-specific issues have the Jira component "LINQ3" in this CSHARP project.
Now I find issues under "Builders", "linq", ... (Jira supports multiple components, so you could also just add "LINQ3" where more appropriate)

This allows all to readily track LINQ3 issues project = CSHARP AND resolution = Unresolved AND component = LINQ3 ORDER BY priority DESC, updated DESC

Comment by Sys Ops [ 18/Oct/23 ]

mirek@rynly.com exactly! robert@mongodb.com is also aware of major breaking change CSHARP-4535  (others reported it, we wasted time testing and investigating why key parts of our application were broken).

Comment by Mirek Kukla [ 18/Oct/23 ]

We would also very much like to see a comprehensive list of breaking changes. We originally discovered CSHARP-4509 when trying migrate to LINQ3 - thank you for the quick fix! Today we attempted to migrate again, but this time we ran into CSHARP-4079.

While we were fortunate enough to catch this breaking change via unit tests, we are now very worried there might be other "minor differences" that break our application (as CSHARP-4079 very much did).

Until such time I'm wary of going to production with LINQ3. Thanks in advance!

Comment by Sys Ops [ 22/Sep/23 ]

robert@mongodb.com, ttutko@outlook.com  Thanks for the clarifications and wholeheartedly support that a list with differences is definitely worth finding and documenting.
robert@mongodb.com do I understand correctly that such a list (or release notes page update) is now actively being worked on? (Do you have any ETA per chance?)

Comment by Robert Stam [ 18/Sep/23 ]

Thanks for your comment. You are correct that I was forgetting a few minor differences. Those are definitely worth finding and documenting. But there shouldn't be very many.

Comment by Thomas Tutko [ 18/Sep/23 ]

robert@mongodb.com Thanks for the update! I appreciate that you guys do not intend to have breaking changes. However, as the original poster of this bug, the explicit error message given in the exception says otherwise in regards to the "$" positional operator as one example that is a breaking change that at least someone who wrote the code was aware of. It's cases like that (if any others exist) that I would like to see clearly documented and listed as a breaking change between v2 and v3. In fact your very own comment on https://jira.mongodb.org/browse/CSHARP-4079 mentions 3 such changes (including the one I ran into). Are you saying you intend to make the old way work too? Thanks for looking into this, your efforts are appreciated.

Comment by Robert Stam [ 18/Sep/23 ]

The primary differences between LINQ2 and LINQ3 are:

  • a number of bugs that exist in LINQ2 do not exist in LINQ3
  • LINQ3 supports more queries than LINQ2 does

We didn't originally plan to have any documentation for what has changed from LINQ2 to LINQ3 because the goal was that everything that works in LINQ2 would also work in LINQ3. Before we released LINQ3 (even before we made it the default) we made sure that all existing tests against LINQ2 also worked against LINQ3.

That being said, since the universe of possible LINQ queries is vast, some users have found scenarios that worked in LINQ2 that didn't work in LINQ3. Our goal is to fix any such issues so that LINQ3 is a strict superset of LINQ2, and there shouldn't be any need for any documentation on the differences between LINQ2 and LINQ3 because there shouldn't be any breaking changes.

One exception to the goal that everything that works in LINQ2 should also work in LINQ3 was that we originally decided to not implement client-side projections in LINQ3 based on the fact that client-side projections can hide potentially large inefficiencies becase it's not always clear how much data will be fetched from the server to be passed to the client-side projection. However, we now plan to add support for client-side projections to LINQ3 based on the number of users that report being affected by this (note that just as in LINQ2, we only plan to support client-side projections with Find, not with Aggregate or AsQueryable). See CSHARP-4763

If you are affected by an open issue that we have not yet fixed, you can still upgrade to the latest version of the driver and configure your settings to continue to use LINQ2, like this:

 

var settings = MongoClientSettings.FromConnectionString("mongodb://localhost");
settings.LinqProvider = LinqProvider.V2;
var client = new MongoClient(settings); 

 

 

Comment by Sys Ops [ 18/Sep/23 ]

A list of breaking changes and known (LINQ3) issues and migration steps would be highly appreciated also by my team. If a comprehensive list is not feasible, than a live document with FAQ, pitfalls would be better than nothing.

At the moment all that I found was this line "The default LinqProvider has been changed to LINQ3." here https://www.mongodb.com/docs/drivers/csharp/current/upgrade/#version-2.19.0-breaking-changes

 

Comment by Lukas Vosyka [ 22/May/23 ]

I'd also love to see more documentation on what exactly has changed between those 2 providers (I guess there is a lot that changed) as well as more detailed examples and how-tos on the official documentation (e.g. I am missing instructions on how to update array ellements within documents see https://www.mongodb.com/docs/drivers/csharp/current/usage-examples/updateOne/ )

 

Comment by James Kovacs [ 11/May/23 ]

Thank you very much, suikevil. We appreciate the effort to file those tickets with repros. We will investigate and get back to you with our findings in those tickets.

Comment by Roberto Pérez [ 11/May/23 ]

Hi james.kovacs@mongodb.com,

 

I have submitted two Jira tickets with example codes and output:

LINQ3 provider: ValueType 'X' of parameterSerializer does not match parameter type 'Y' (where X is a subclass of Y)

https://jira.mongodb.org/browse/CSHARP-4650

 

LINQ3 provider: MongoDB.Driver.Linq.ExpressionNotSupportedException: Expression not supported: 1 in (1 == 1)

https://jira.mongodb.org/browse/CSHARP-4651

 

Regards,

 

R.

Comment by James Kovacs [ 10/May/23 ]

Hi, suikevil,

I'm sorry to hear that you've encountered so many issues in upgrading to 2.19.1 and LINQ3. LINQ3 is a more full-featured LINQ provider also fixing many longstanding bugs in LINQ2 but the transition hasn't been as seamless as desired. LINQ2 will continue to be supported until the 3.X driver series, which is still a ways away. We will also continue to aggressively investigate and fix any reported LINQ3 issues such as the ones that you encountered.

Of the three issues, the third was an intentional change (CSHARP-4079) to replace a magic number (-1) with a fluent method (FirstMatchingElement). This also allowed us to support AllElements and AllMatchingElements. Admittedly we could have documented this better and this ticket (CSHARP-4618) intends to do just that.

The other two issues that you mention are not known issues. We would greatly appreciate if you could file CSHARP tickets with small self-contained repros so that we can investigate further. Thank you in advance.

Sincerely,
James

Comment by Roberto Pérez [ 10/May/23 ]

My lifeserver: https://github.com/mongodb/mongo-csharp-driver/releases/tag/v2.19.0

 
var connectionString = "mongodb://localhost";
var clientSettings = MongoClientSettings.FromConnectionString(connectionString);
clientSettings.LinqProvider = LinqProvider.V2;
var client = new MongoClient(clientSettings);
 
 

Comment by Roberto Pérez [ 10/May/23 ]

Yesterday, I tried to migrate from 2.18 (a vulnerability in one of the packages was detected) to 2.19.1 and now our solution explodes in many points that used to worked perfectly.

  • MongoDB.Driver.Linq.ExpressionNotSupportedException: Expression not supported: 2 in (2 == 3) because it was not possible to determine how to serialize the constant.
  • ValueType 'X' of parameterSerializer does not match parameter type 'Y' (where X is a subclass of Y)
  • Expression not supported: d.xx.get_Item(-1) because negative indexes are not valid. To use the positional operator $ use FirstMatchingElement instead of an index value of -1.

 

I am still looking for samples to address these breaking changes and but I couldn't find any. We uses lambdas for everything and I am afraid that these changes break them

Generated at Wed Feb 07 21:48:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.