[SERVER-76641] Time Series collection will double the calculation when using $sum in aggregate Created: 28/Apr/23  Updated: 20/Jun/23  Resolved: 20/Jun/23

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 6.0.5
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: 解海 N/A Assignee: Chris Kelly
Resolution: Cannot Reproduce Votes: 0
Labels: aggregate
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File 微信图片_20230428112716.png    
Operating System: ALL
Participants:

 Description   

Example:

I have the following documents

 

// 759
{
    "symbol": "MA309",
    "volume": NumberInt("14"),
    "timestamp": ISODate("2023-04-28T09:57:59.000Z")
}// 760
{
    "symbol": "rb2310",
    "volume": NumberInt("1"),
    "timestamp": ISODate("2023-04-28T09:57:59.000Z")
}// 761
{
    "symbol": "MA309",
    "volume": NumberInt("6"),
    "timestamp": ISODate("2023-04-28T09:57:59.001Z")
}// 762
{
    "symbol": "rb2310",
    "volume": NumberInt("52"),
    "timestamp": ISODate("2023-04-28T09:57:59.5Z")
}// 763
{
    "symbol": "MA309",
    "volume": NumberInt("7"),
    "timestamp": ISODate("2023-04-28T09:58:00.000Z")
}// 764
{
    "symbol": "MA309",
    "volume": NumberInt("226"),
    "timestamp": ISODate("2023-04-28T09:58:00.001Z")
}// 765
{
    "symbol": "rb2310",
    "volume": NumberInt("17"),
    "timestamp": ISODate("2023-04-28T09:58:00.5Z")
}// 766
{
    "symbol": "MA309",
    "volume": NumberInt("108"),
    "timestamp": ISODate("2023-04-28T09:58:01.000Z")
}// 767
{
    "symbol": "rb2310",
    "volume": NumberInt("4"),
    "timestamp": ISODate("2023-04-28T09:58:01.000Z")
}// 768
{
    "symbol": "rb2310",
    "volume": NumberInt("19"),
    "timestamp": ISODate("2023-04-28T09:58:01.5Z")
}// 769
{
    "symbol": "MA309",
    "volume": NumberInt("2"),
    "timestamp": ISODate("2023-04-28T09:58:02.000Z")
}// 770
{
    "symbol": "rb2310",
    "volume": NumberInt("1"),
    "timestamp": ISODate("2023-04-28T09:58:02.000Z")
} 

Aggregate using the following

 

db.getCollection("ticks").aggregate([{
    $group: {
        _id: {
           symbol: "$symbol",
           minute: {
               $dateToString: {
                  format: "%Y-%m-%d %H:%M",
                  date: "$timestamp",                        
                }                   
            },             
        },
        volume: {
            $sum: "$volume"
        }
    }
}])

If $sum turns out to be 10, then there is a 1% chance that the incidental calculation will be 20.

 

 



 Comments   
Comment by Chris Kelly [ 20/Jun/23 ]

We haven’t heard back from you for some time, so I’m going to close this ticket. If this is still an issue for you, please provide additional information and we will reopen the ticket.

Comment by Chris Kelly [ 06/Jun/23 ]

Hi 50083000@qq.com

We still need additional information to diagnose the problem. If this is still an issue for you, can you please provide additional data? Specifically:

For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to the ticket:

  • the mongod logs
  • the $dbpath/diagnostic.data directory (the contents are described here)
Comment by Chris Kelly [ 19/May/23 ]

Thanks for the extra detail 50083000@qq.com!

 

Some notes:

  • I ran a test that simply inserts, and performs the aggregation repeatedly in a single thread with no chance of reproducing the issue on 6.0.4 on Ubuntu 20.04 / WSL2 on thousands of records with multiple threads
  • I was only able to reproduce changing volume sizes from their aggregation while concurrently performing inserts, which is expected behavior.
    • Is there a race condition in your code around this point?
  • performing concurrent runs of the aggregation were not reproducing the issue without changing the data.

Can you please provide additional data? Specifically:

For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to the ticket:

  • the mongod logs
  • the $dbpath/diagnostic.data directory (the contents are described here)

Additionally, if you have a specific test driver that is reproducing the issue, that would be appreciated (or any other information you can provide around thee issue)

Thanks!

Chris

Comment by 解海 N/A [ 10/May/23 ]
Mongodb Version OS node
6.0.4 Community Debian 11(bullseye) single

If you need more information you can tell me.

Thanks!

Comment by Chris Kelly [ 09/May/23 ]

Hi 50083000@qq.com,

Thanks for your report. Just for completeness, can we also get the info regarding your MongoDB version and environment? (OS, architecture, how many nodes, etc)

Thanks!

Comment by 解海 N/A [ 28/Apr/23 ]

The problem described only occurs with some probability when a large amount of data is aggregated.

Comment by 解海 N/A [ 28/Apr/23 ]

The isuss did not appear after the change to a normal collection.

Generated at Thu Feb 08 06:33:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.