[CSHARP-4468] LINQ V3 SelectMany + GroupBy results with redundant $push within $group Created: 24/Dec/22  Updated: 26/Jan/23  Resolved: 12/Jan/23

Status: Closed
Project: C# Driver
Component/s: LINQ3
Affects Version/s: 2.18.0
Fix Version/s: 2.19.0

Type: Improvement Priority: Unknown
Reporter: Robert Stam Assignee: Robert Stam
Resolution: Done Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Details copied from: https://www.mongodb.com/community/forums/t/linq-v3-selectmany-groupby-results-with-redundant-push-within-group/205648

After switching to LinqV3 some reports starts too fail because of Mongo exception:

Command aggregate failed: Exceeded memory limit for $group, but didn’t allow external sort. Pass allowDiskUse:true to opt in

Seems like the reason is the query generated with v3 which contains unnecessary $push within $group phase, below is sample code to reproduce the issue:

using System;
using System.Linq;
using System.Security.Authentication;
using MongoDB.Driver;
using MongoDB.Driver.Linq;
 
var settings = MongoClientSettings.FromUrl(new MongoUrl("mongodb://localhost:27017/test"));
settings.SslSettings = new SslSettings {EnabledSslProtocols = SslProtocols.Tls12};
settings.LinqProvider = LinqProvider.V3;
var mongoClient = new MongoClient(settings);
var mongoDatabase = mongoClient.GetDatabase("test");
var collection = mongoDatabase.GetCollection<OrderDao>("test");
 
var query1 = collection
    .AsQueryable()
    .SelectMany(i => i.Lines)
    .GroupBy(l => l.ItemId)
    .Select(g => new ItemSummary
    {
        Id = g.Key,
        TotalAmount = g.Sum(l => l.TotalAmount)
    });
 
var query1txt = query1.ToString();
 
Console.WriteLine(query1txt);
Console.WriteLine(query1txt.Contains("$push") ? "Uses $push :(" : "No $push here, hurray!");
 
var query2 = collection
    .AsQueryable()
    .GroupBy(l => l.Id)
    .Select(g => new ItemSummary
    {
        Id = g.Key,
        TotalAmount = g.Sum(l => l.TotalAmount)
    });
 
var query2txt = query2.ToString();
 
Console.WriteLine(query2txt);
Console.WriteLine(query2txt.Contains("$push") ? "Uses $push :(" : "No $push here, hurray!");
 
public class OrderDao
{
    public OrderLineDao[] Lines { get; set; }
    
    public decimal TotalAmount { get; set; }
    public Guid Id { get; set; }
}
 
public class OrderLineDao
{
    public decimal TotalAmount { get; set; }
    public Guid ItemId { get; set; }
}
 
public class ItemSummary
{
    public Guid Id { get; set; }
    public decimal TotalAmount { get; set; }
}

Output when executed with LinqV2:

aggregate([{ "$unwind" : "$Lines" }, { "$project" : { "Lines" : "$Lines", "_id" : 0 } }, { "$group" : { "_id" : "$Lines.ItemId", "__agg0" : { "$sum" : "$Lines.TotalAmount" } } }, { "$project" : { "Id" : "$_id", "TotalAmount" : "$__agg0", "_id" : 0 } }])
 
No $push here, hurray!
 
aggregate([{ "$group" : { "_id" : "$_id", "__agg0" : { "$sum" : "$TotalAmount" } } }, { "$project" : { "Id" : "$_id", "TotalAmount" : "$__agg0", "_id" : 0 } }])
 
No $push here, hurray!

Output when executed with LinqV3:

test.test.Aggregate([{ "$project" : { "_v" : "$Lines", "_id" : 0 } }, { "$unwind" : "$_v" }, { "$group" : { "_id" : "$_v.ItemId", "_elements" : { "$push" : "$_v" } } }, { "$project" : { "_id" : "$_id", "TotalAmount" : { "$sum" : "$_elements.TotalAmount" } } }])
 
Uses $push :(
 
test.test.Aggregate([{ "$group" : { "_id" : "$_id", "__agg0" : { "$sum" : "$TotalAmount" } } }, { "$project" : { "_id" : "$_id", "TotalAmount" : "$__agg0" } }])
 
No $push here, hurray!



 Comments   
Comment by Marek Olszewski [ 26/Jan/23 ]

Any ETA for a nuget release?

Comment by Githook User [ 12/Jan/23 ]

Author:

{'name': 'rstam', 'email': 'robert@robertstam.org', 'username': 'rstam'}

Message: CSHARP-4468: LINQ V3 SelectMany + GroupBy results with redundant $push within $group.
Branch: master
https://github.com/mongodb/mongo-csharp-driver/commit/396830ccf8312238cce269774e5955bcaeda7857

Generated at Wed Feb 07 21:48:19 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.