[CSHARP-4507] 2.19 Projection provides unexpected results on 4.2 and lower Created: 31/Jan/23 Updated: 27/Oct/23 Resolved: 27/Jun/23 |
|
| Status: | Closed |
| Project: | C# Driver |
| Component/s: | LINQ3 |
| Affects Version/s: | 2.19.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Unknown |
| Reporter: | Nick Judson | Assignee: | James Kovacs |
| Resolution: | Works as Designed | Votes: | 2 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Documentation Changes Summary: | 1. What would you like to communicate to the user about this feature? |
| Description |
|
Please see this post: https://www.mongodb.com/community/forums/t/2-19-breaks-projections/211242/8
|
| Comments |
| Comment by James Kovacs [ 05/Apr/23 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
You are correct, nick@innsenroute.com. Find maps to the find command, which takes a variety of parameters including filter (aka the match clause or predicate), sort, projection, skip, limit, and more. It has defined semantics when all these fields are supplied. For example, it will filter and sort the documents prior to the skip/limit. Aggregate.Match maps to the aggregate command, which accepts an aggregation pipeline. This is a much more flexible and expressive API. You can skip documents before or after matching them. This would change the semantics of the query and also the ability to use an index. For example, if you want to skip 1000 documents prior to a match, you can't use an index because the index has no notion of how many unindexed documents appeared before the first index entry. MongoDB's query planner can perform a variety of optimizations, such as logically pulling match stages earlier in the pipeline, but it cannot violate the semantics of the aggregation pipeline as written. As the adage goes, with great power comes great responsibility. The aggregation pipeline provides a lot of flexibility to express your query intentions, but it also allows you to write queries that are difficult/impossible for the query planner to optimize. With any query, you can call query.ToString() to view the MQL that will be sent to the server. (You can also install the MongoDB Analyzer NuGet package, which will show you the MQL as a tooltip.) You can use the MQL to run an explain plan in the mongosh shell to examine the query plan. MongoDB Atlas also includes the MongoDB Atlas Performance Advisor, which will display slow queries and provide index recommendations. It is also worth reading Optimizing MongoDB Compound Indexes. Hope that helps. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Nick Judson [ 04/Apr/23 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
james.kovacs@mongodb.com - quick question regarding replacement of .Find with .Aggregate.Match: Is it correct that `.Find` does not care about the order of each subsequent expression? Ie, the order of `.order`, `.project`, `.limit` etc. doesn't matter? Is it correct that for `.Aggregate().Match`, the order does matter? I noticed some performance regressions after doing a blanket replace of `.Find` with `.Aggregate().Match`, and traced it back to the order of the expressions when moving away from `.Find`. Thanks. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by José Massada [ 17/Feb/23 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I was working around the issue by projecting to an anonymous class but I've also found that `[BsonId]` isn't taken into account and so the return for the _id field will be the default value. Not sure if I should open a new issue.
Edit: works with Aggregate + Match
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Alessandro Giorgetti [ 16/Feb/23 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
I also noticed the same issue event with MongoDB 6.0.4. To reproduce the issue modify the sample code above in this way:
Add a new property to MongoMessage and leave it uninitialized:
Add the following code to the main test routine:
When you run the program you can see the very same error even on the most recent version of MongoDB | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Remco Ros [ 13/Feb/23 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
This also affects users of "Azure Cosmos DB for MongoDB" using version 2.19.0 of MongoDB.Driver. Have not verified if using an aggregation pipeline (as suggested) will fix this for Cosmos, as we reverted to 2.18.0 for now, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Nick Judson [ 01/Feb/23 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Hi James, Thanks for the explanation. I have switched over to using the Fluent Aggregate and can confirm that fixes the issue. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by James Kovacs [ 31/Jan/23 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
[Cross-posting my Community Forums response for easier reference.] Hi, Nick, Thank you for your patience and thank you for filing To answer your first question, MongoDB 4.2 is still a supported server version though it will reach end-of-life in April 2023. See our support policy for full details. Now let's discuss the root cause of this issue. The problem stems from the renaming of the fields in your LINQ query. The following code will display the MQL sent to the server. The same MQL is sent to MongoDB regardless of the server version.
The resulting MQL is:
The second argument to the `find` command is the projection. Note the syntax "CsharpFieldName": "$databaseFieldName". For example "GridFsObjectId": "$grid". This is where the problem lies. In MongoDB 4.4 and later, you could use this $fieldName syntax to rename fields in projections - whether those projections were part of a find operation or an aggregation pipeline. However in MongoDB 4.2 and earlier, you could only use this syntax in aggregation pipelines, but not in find projections. Find projections only allowed the older, simpler syntax of fieldName: 1 to include a field. This is further complicated by the fact that for backwards compatibility, fieldName: VALUE where VALUE was truthy in the JavaScript sense. This meant that 0, false, null, and undefined are interpreted as false and pretty much everything else is interpreted as true. Thus MongoDB 4.2 (using find) sees "GridFsObjectId": "$grid" as "GridFsObjectId": true. Since there is no field in the document named GridFsObjectId, the field is omitted leading to the observed behaviour. In MongoDB 4.4, we enhanced the find projection to use the same syntax as the aggregation pipeline. Thus MongoDB 4.4 sees "GridFsObjectId": "$grid" as "GridFsObjectId": "$grid" and correctly renames the field grid (in the database) to GridFsObjectId in the returned document. Possible workarounds for this issue (in no particular order):
The first two should be self-explanatory. I will note that LINQ3 has greatly enhanced capabilities including support for new aggregation features. LINQ2 will be deprecated in an upcoming version and removed in the 3.0.0 driver. We have not announced a public timeline for the 3.0.0 driver. Refactoring to use Fluent Aggregation is probably the most straightforward. You can use the same FilterDefinition<> and ProjectionDefinition<,> as you are currently using with Find/Project. Rather than coll.Find(filter).Project(projection) you would instead use coll.Aggregate().Match(filter).Project(projection).
The resulting MQL produces an aggregation pipeline rather than a find command, but the query results are the same:
This aggregation pipeline - including the projection - will work on even very old server versions. I tested it on MongoDB 3.6 and it produced the correct results. Please follow Sincerely, | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Comment by Nick Judson [ 31/Jan/23 ] | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Here is the repro (copied from the linked post):
And here is the output (<= v4.2):
>= 4.4
|