[CSHARP-4852] Linq3Provider does not deal correctly with non-existing objects Created: 21/Nov/23  Updated: 19/Dec/23  Resolved: 19/Dec/23

Status: Closed
Project: C# Driver
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Unknown
Reporter: Sjoerd Grevelink Assignee: Robert Stam
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to CSHARP-4858 Provide a way to test if a field exis... Closed
Documentation Changes Summary:

1. What would you like to communicate to the user about this feature?
2. Would you like the user to see examples of the syntax and/or executable code and its output?
3. Which versions of the driver/connector does this apply to?


 Description   

Summary

Especially notable during projection, if you check whether a field exists or not by doing a null-check, or by using string.IsNullOrEmpty or similar, it will return as not null when the field, or a parent of that field does not exist.

This is a breaking change in version 2.22.0 of the driver, coming from 2.18.0.

How to Reproduce

Add this to a collection:

 

{
  "_id": { "$oid": "000000000000000000000001" },
  "Name": "Example Name",
  "SecretValue": "Some Secret Value"
}

With the following classes:

 

 

public class MainClass
{
    [BsonId]
    public ObjectId Id { get; set; }
    public string Name { get; set; }
    public string SecretValue { get; set; }
    public OptionalClass OptionalData { get; set; }
}
 
public class OptionalClass
{
   public string PublicData { get; set; }
   public string PrivateData { get; set; }
}
 
public class ProjectedMainClassWithoutSecretData
{
    [BsonId]
    public ObjectId Id { get; set; }
    public string Name { get; set; }
    public ProjectedOptionalClassWithoutSecretData OptionalData { get; set }
    public bool HasOptionalPrivateData { get; set; }
}
 
public class ProjectedOptionalClassWithoutSecretData
{
    public string PublicData { get; set; }
}

Then do a find like so:

 

async Task<ProjectedMainClassWithoutSecretData> GetPublicData(IMongoCollection<MainClass> collection, ObjectId itemId)
{
    var filter = Builders<MainClass>.Filter.Eq(x => x.Id, itemId);
    var projection = Builders<MainClass>.Projection.Expression(main => new ProjectedMainClassWithoutSecretData() {
      Id = main.Id,
      Name = main.Name,
      OptionalData = main.OptionalData != null ? new ProjectedOptionalClassWithoutSecretData() {
         PublicData = main.OptionalData.PublicData
      } : null,
      HasOptionalPrivateData = !string.IsNullOrEmpty(main.OptionalData.PrivateData)
    });
    var result = await collection.Find(filter).Project(projection).FirstOrDefaultAsync();
    return result;
}

You will find that the result will have OptionalData set, and HasOptionalPrivateData set to true, where null and false were expected.

 

 

Additional Background

From what I can tell, there is no easy work-around to check for unset fields, and when dealing with larger classes, manual projection is not maintainable.

Please provide any additional background information that may be helpful in diagnosing the bug.



 Comments   
Comment by Robert Stam [ 19/Dec/23 ]

> If you check the Render() of the projection, you will see that it renders these as $ne / $eq for != null / == null and $in with "" and null for string.IsNullOrEmpty. This however misses the "$UNSET" that is needed for non-existence of the field.

I explained in point 3 above that we chose not to require the MQL resulting from translation C# LINQ expressions to handle the case of missing fields.

If you think a field might be missing you should account for that in your query.

Comment by Sjoerd Grevelink [ 29/Nov/23 ]

I would also like to suggest:

bool MongoDB.IsMissingOrDefault<T>(T field, T defaultValue = default) 

Comment by Sjoerd Grevelink [ 29/Nov/23 ]

Thanks for looking into this, Robert, and thanks for the update.

I would personally prefer an option to follow the settings on the BSON class map, but the biggest issue is that there is currently no way to test for the missing fields. I look forward to your addition of those helper methods.

Comment by Robert Stam [ 28/Nov/23 ]

Thank you for reporting this issue. I am able to reproduce the behavior you have reported.

There are several related issues going on here:

1. Client-side vs server-side projection

When using LINQ2 the Find Projection is executed client side, but when using LINQ3 the projection is executed server-side.

A missing OptionalData field is deserialized as null so when executing client-side the predicate "main.OptionalData == null" will be true when OptionalData is missing.

But when executing server side "main.OptionalData == null" is translated as "

{ $eq : ["$OptionalData", null] }

" which according to server semantics for $eq is not true for a missing field.

2. Client-side projections should be explicit when using LINQ3

You could convert your Find to use explicit client-side projection instead of server-side projection. This is done by writing the query in a way that separates what happens server-side from what happens client-side.

Your query could be re-written like this:

var results = collection.Find(filter) 
    .ToEnumerable() // everything after this line executes client-side
    .Select(main => new ProjectedMainClassWithoutSecretData()
    {
        Id = main.Id,
        Name = main.Name,
        OptionalData = main.OptionalData != null ?
            new ProjectedOptionalClassWithoutSecretData()
            {
                PublicData = main.OptionalData.PublicData
            } :
            null,
        HasOptionalPrivateData = !string.IsNullOrEmpty(main.OptionalData?.PrivateData)
    })
    .ToList();

3. We chose to make testing for missing fields the application's responsibility in LINQ queries

The reason we made this choice is to benefit applications that don't have missing fields. The query translations will be simpler and will execute faster.

If the application expects fields to be missing it should test for that explicitly and decide what a missing field means.

4. Unfortunately we have not yet provided a way to test if a field is missing

I will find or create a JIRA ticket for this functionality and link this ticket to it. Most likely this will involve custom static methods that our LINQ3 provider will recognize and translate.

The idea is that instead of writing:

main.OptionalData == null // is false for a missing field

You would write something like:

MongoDB.IsMissing(main.OptionalData)

or:

MongoDB.IsNullOrMissing(main.OptionalData)

Comment by Sjoerd Grevelink [ 21/Nov/23 ]

If you check the Render() of the projection, you will see that it renders these as $ne / $eq for != null / == null and $in with "" and null for string.IsNullOrEmpty. This however misses the "$UNSET" that is needed for non-existence of the field.

Comment by PM Bot [ 21/Nov/23 ]

Hi sgrevelink@gmail.com, thank you for reporting this issue! The team will look into it and get back to you soon.

Generated at Wed Feb 07 21:49:34 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.