[CSHARP-4493] "An object representing an expression must have exactly one field" when querying by Linq query with .ToLower() Created: 25/Jan/23  Updated: 28/Oct/23  Resolved: 23/May/23

Status: Closed
Project: C# Driver
Component/s: LINQ3
Affects Version/s: 2.18.0
Fix Version/s: 2.19.2

Type: Bug Priority: Unknown
Reporter: Sorin Pochtar Assignee: Oleksandr Poliakov
Resolution: Fixed Votes: 0
Labels: triage
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Backwards Compatibility: Fully Compatible
Documentation Changes: Not Needed
Documentation Changes Summary:

1. What would you like to communicate to the user about this feature?
2. Would you like the user to see examples of the syntax and/or executable code and its output?
3. Which versions of the driver/connector does this apply to?


 Description   

Summary

"An object representing an expression must have exactly one field" exception occurs when querying by Linq query with multiple Any/Contains conditions, each of which having .ToLower() operator, and conditions combined by &&.

 

C# Driver version: 2.18.0

Server version: 6.0.2, standalone, hosted on Docker container
LinqProvider: LinqProvider.V3

How to Reproduce

Build the following Linq query and use Find(...) driver's method to query database:

 

Expression<Func<Customer, bool>> filterByAddressAndEmails =
     c => c.Emails.Any(ce => emails.Contains(ce.Email.ToLower())) &&
          addresses.Contains(c.Address.FullAddress.ToLower());

 

where Customer class is:

public class Customer
{      
    [BsonId] 
    [BsonIgnoreIfDefault] 
    [BsonRepresentation(BsonType.String)] 
    public ObjectId Id { get; set; }     
    public AddressMetadata Address { get; set; }
    public IList<EmailMetadata> Emails { get; set; }
    ...
    public sealed record AddressMetadata
    {
        ...
        public string FullAddress { get; set; }
        ...
    }
 
    public sealed record EmailMetadata
    {
        ...
        public string Email { get; set; }
        ..
    }
} 

and variables are:

 

 

string[] emails = new[] { "email4@test.net" };
string[] addresses = new[] { "495 pacific street plymouth, ma 02360" };

 

the following exception is generated:

Command find failed: An object representing an expression must have exactly one field: 
    $in: [ 
        { 
            $toLower: "$Address.FullAddress" 
        }, 
        [ "495 pacific street plymouth, ma 02360"
    ], 
    $anyElementTrue: 
    { 
        $map: 
            { 
                input: "$Emails"
                as: "ce"
                in: 
                    { 
                        $in: [ 
                            { $toLower: "$$ce.Email" }, 
                            [ "email4@test.net"
                        ] 
                    } 
            } 
    } 
} 

 

 

Additional Background

As a workaround to the issue, I had to split the query into two, one of which includes OR operator:

Expression<Func<Customer, bool>> filterByAddressAndEmails =
    c => c.Emails.Any(ce => emails.Contains(ce.Email.ToLower()));
 
Expression<Func<Customer, bool>> filterByAddress =
    c => addresses.Contains(c.Address.FullAddress.ToLower()) ||
 addresses.Contains(c.Address.FullAddress);
 
filterByFullAddressAndEmails = filterByFullAddressAndEmails.And(filterByFullAddress); 
 
where .And(...) is an extension method which can combine Linq queries.

Also, if the original query is modified that one of .ToLower() methods are removed, the query does not generate an exception, but does not return a correct result, because I need case insensitive match;

 



 Comments   
Comment by Githook User [ 24/May/23 ]

Author:

{'name': 'Oleksandr Poliakov', 'email': '31327136+sanych-sun@users.noreply.github.com', 'username': 'sanych-sun'}

Message: CSHARP-4493: Fix incorrect flattening of $expr in $and operator (#1072)
Branch: v2.19.x
https://github.com/mongodb/mongo-csharp-driver/commit/5a973c22eef58fd7049789afbc826461eef9476d

Comment by Githook User [ 23/May/23 ]

Author:

{'name': 'Oleksandr Poliakov', 'email': '31327136+sanych-sun@users.noreply.github.com', 'username': 'sanych-sun'}

Message: CSHARP-4493: Fix incorrect flattening of $expr in $and operator (#1072)
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/ecb7d273c13691d29c04cf5fc16c72f8d6ecb022

Comment by James Kovacs [ 15/Feb/23 ]

Thank you for the additional details. We have successfully reproduced the bug and it is subtle...

Using ToLower() necessitates the use of $expr to call the server-side $toLower function. Since you have two clauses in your filter both using $expr, you end up with a filter that looks roughly like this:

{$expr: expr1}, {$expr: expr2}

where expr1 is your any emails clause and expr2 is your contains address clause.

When you call filter & empty, we optimize the predicate flattening the $and. And this is the root cause of the problem. We flatten:

{$expr: expr1}, {$expr: expr2}

into

{$expr: {expr1, expr2} }

In many/most cases in MQL, {{ expr1, expr2 }} is equivalent to $and: [expr1, expr2], but $expr requires a single argument, which cannot be an array. Thus the server throws an error when we send it this incorrectly simplified MQL.

Now why doesn't this happen when you don't filter & empty? Because we don't attempt to simplify the filter and send {{ {$expr: expr1}, {$expr: expr2} }} to the server.

Why doesn't this happen when you omit ToLower()? Because we don't need to use $expr to call $toLower and we don't combine expressions in an invalid way.

We will include a fix for this issue in an upcoming release. Please watch this ticket for further updates.

Below is a repro of the issue:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using MongoDB.Bson;
using MongoDB.Driver;
 
string[] emails = { "email4@test.net" };
string[] addresses = { "495 pacific street plymouth, ma 02360" };
 
#pragma warning disable CA1304
Expression<Func<Customer, bool>> filterByAddressAndEmails =
    c => c.Emails.Any(ce => emails.Contains(ce.Email.ToLower())) &&
         addresses.Contains(c.Address.FullAddress.ToLower());
 
FilterDefinition<Customer> emptyPredicate = Builders<Customer>.Filter.Empty;
#pragma warning restore CA1304
 
var client = new MongoClient();
var db = client.GetDatabase("test");
var coll = db.GetCollection<Customer>("customers");
 
// No & operator. Resulting MQL is valid.
var findQueryWithoutEmpty = coll.Find(filterByAddressAndEmails, null);
Console.WriteLine(findQueryWithoutEmpty);
// The following iterates successfully
foreach (var customer in findQueryWithoutEmpty.ToList())
{
    Console.WriteLine(customer.Address.FullAddress);
}
 
// & operator incorrectly flattens {$expr: expr1}, {$expr: expr2}
// into {$expr: {expr1, expr2} } which is invalid MQL.
// It should be {$expr: { $and: [expr1, expr2] } }
var findQueryWithEmpty = coll.Find(filterByAddressAndEmails & emptyPredicate, null);
Console.WriteLine(findQueryWithEmpty);
// The following throws Command find failed: An object representing an expression must have exactly one field
foreach (var customer in findQueryWithEmpty.ToList())
{
    Console.WriteLine(customer.Address.FullAddress);
}
 
public class Customer
{
    public ObjectId Id { get; set; }
    public AddressMetadata Address { get; set; }
    public IList<EmailMetadata> Emails { get; set; }
}
 
public sealed record AddressMetadata
{
    public string FullAddress { get; set; }
}
 
public sealed record EmailMetadata
{
    public string Email { get; set; }
}

Comment by Sorin Pochtar [ 28/Jan/23 ]

Hi @James,

Sorry, I have skipped one detail which I considered unimportant.

Our framework is adding an empty filter under the hood (someAdditionalPredicate is my example),

so the actual query will look like:

 

Expression<Func<Customer, bool>> filterByAddressAndEmails =
    c => c.Emails != null && 
        c.Emails.Any(ce => emails.Contains(ce.Email.ToLower())) &&
         addresses.Contains(c.Address.FullAddress.ToLower());
 
FilterDefinition<Customer> someAdditionalPredicate = Builders<Customer>.Filter.Empty;
 
List<Customer> customers = 
    customersCollection
    .Find(filterByAddressAndEmails & someAdditionalPredicate, null)
    .ToList(); 

As you can see, I added

c.Emails != null && 

predicate, but it does not help.

The same as previously, if I remove one of .ToLower() the query does not generate an exception.

 

 

 

 

 

 

Comment by James Kovacs [ 28/Jan/23 ]

Hi, sorinpochtar@gmail.com,

Thank you for reaching out to us about this issue. We have been unsuccessful in reproducing the exception message that you encountered. Here is the self-contained repro code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Linq.Expressions;
using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoDB.Driver;
 
string[] emails = new[] { "email4@test.net" };
string[] addresses = new[] { "495 pacific street plymouth, ma 02360" };
 
#pragma warning disable CA1304
Expression<Func<Customer, bool>> filterByAddressAndEmails =
    c => c.Emails.Any(ce => emails.Contains(ce.Email.ToLower())) &&
         addresses.Contains(c.Address.FullAddress.ToLower());
#pragma warning restore CA1304
 
var client = new MongoClient();
var db = client.GetDatabase("test");
var coll = db.GetCollection<Customer>("customers");
 
var query = coll.AsQueryable().Where(filterByAddressAndEmails);
Console.WriteLine(query);
foreach (var customer in query.ToList())
{
    Console.WriteLine(customer.Address.FullAddress);
}
 
public class Customer
{
    [BsonId]
    [BsonIgnoreIfDefault]
    [BsonRepresentation(BsonType.String)]
    public ObjectId Id { get; set; }
 
    public AddressMetadata Address { get; set; }
    public IList<EmailMetadata> Emails { get; set; }
}
 
public sealed record AddressMetadata
{
    public string FullAddress { get; set; }
}
 
public sealed record EmailMetadata
{
    public string Email { get; set; }
}

When running this repro against a customers collection containing documents without an Emails array, I ran into a similar though not identical issue:

Command aggregate failed: PlanExecutor error during aggregation :: caused by :: $anyElementTrue's argument must be an array, but is null.

I was able to resolve this by adding the condition c.Emails != null && to the beginning of the predicate.

Please try the repro provided above and let us know if there is additional code or particular documents in your collection that is causing it to fail in your use case.

Sincerely,
James

Generated at Wed Feb 07 21:48:23 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.