[SERVER-61029] Inconsistent behavior for time series collections when using insertMany Created: 27/Oct/21  Updated: 04/Nov/21  Resolved: 04/Nov/21

Status: Closed
Project: Core Server
Component/s: None
Affects Version/s: 5.0.3-rc0
Fix Version/s: None

Type: Bug Priority: Major - P3
Reporter: razvan virjgohe Assignee: Edwin Zhou
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates SERVER-60894 Time Series collections bucket re-ope... Closed
Operating System: ALL
Steps To Reproduce:

1. create collection 

db.createCollection(
    'test',
    {
       timeseries: {
          timeField: "date",
          metaField: "metadata",
          granularity: "minutes"
       },
    }
)

2. insert events, all items are ordered except the first 2

db.test.insertMany([
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.657497771333402,
        "date" : ISODate("2021-10-27T13:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.630711479160949,
        "date" : ISODate("2021-10-25T19:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.907835596255776,
        "date" : ISODate("2021-10-28T01:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.478167641630244,
        "date" : ISODate("2021-10-28T07:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.734791557369546,
        "date" : ISODate("2021-10-28T13:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.355601537158719,
        "date" : ISODate("2021-10-28T19:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.0392149962590065,
        "date" : ISODate("2021-10-29T01:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.715255277128178,
        "date" : ISODate("2021-10-29T07:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.912639155007598,
        "date" : ISODate("2021-10-29T13:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.877510759928182,
        "date" : ISODate("2021-10-29T19:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.610814010052608,
        "date" : ISODate("2021-10-27T13:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.51848041136916,
        "date" : ISODate("2021-10-27T19:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.639634772233867,
        "date" : ISODate("2021-10-28T01:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.975665115747611,
        "date" : ISODate("2021-10-28T07:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.941601991088971,
        "date" : ISODate("2021-10-28T13:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.651105330652535,
        "date" : ISODate("2021-10-28T19:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.725467962749386,
        "date" : ISODate("2021-10-29T01:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.694043071407878,
        "date" : ISODate("2021-10-29T07:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.237759276862083,
        "date" : ISODate("2021-10-29T13:47:52.883Z")
    },
    {
        "metadata" : {
            "y" : "a"
        },
        "value" : 0.815200486877654,
        "date" : ISODate("2021-10-29T19:47:52.883Z")
    }
]); 

3. query the buckets collection

db.getCollection('system.buckets.test').find({})
/* 1 */
{
    "_id" : ObjectId("6176feb032850201c3dc46cf"),
    "control" : {
        "version" : 1,
        "min" : {
            "_id" : ObjectId("617958a1d21593e225bfa4d6"),
            "value" : 0.630711479160949,
            "date" : ISODate("2021-10-25T19:00:00.000Z")
        },
        "max" : {
            "_id" : ObjectId("617958a1d21593e225bfa4d6"),
            "value" : 0.630711479160949,
            "date" : ISODate("2021-10-25T19:47:52.883Z")
        }
    },
    "meta" : {
        "y" : "a"
    },
    "data" : {
        "_id" : {
            "0" : ObjectId("617958a1d21593e225bfa4d6")
        },
        "value" : {
            "0" : 0.630711479160949
        },
        "date" : {
            "0" : ISODate("2021-10-25T19:47:52.883Z")
        }
    }
}/* 2 */
{
    "_id" : ObjectId("61794d5032850201c3dc46ce"),
    "control" : {
        "version" : 1,
        "min" : {
            "_id" : ObjectId("617958a1d21593e225bfa4d5"),
            "value" : 0.657497771333402,
            "date" : ISODate("2021-10-27T13:00:00.000Z")
        },
        "max" : {
            "_id" : ObjectId("617958a1d21593e225bfa4d5"),
            "value" : 0.657497771333402,
            "date" : ISODate("2021-10-27T13:47:52.883Z")
        }
    },
    "meta" : {
        "y" : "a"
    },
    "data" : {
        "_id" : {
            "0" : ObjectId("617958a1d21593e225bfa4d5")
        },
        "value" : {
            "0" : 0.657497771333402
        },
        "date" : {
            "0" : ISODate("2021-10-27T13:47:52.883Z")
        }
    }
}/* 3 */
{
    "_id" : ObjectId("61794d5032850201c3dc46d2"),
    "control" : {
        "version" : 1,
        "min" : {
            "_id" : ObjectId("617958a1d21593e225bfa4df"),
            "value" : 0.51848041136916,
            "date" : ISODate("2021-10-27T13:00:00.000Z")
        },
        "max" : {
            "_id" : ObjectId("617958a1d21593e225bfa4e2"),
            "value" : 0.975665115747611,
            "date" : ISODate("2021-10-28T07:47:52.883Z")
        }
    },
    "meta" : {
        "y" : "a"
    },
    "data" : {
        "_id" : {
            "0" : ObjectId("617958a1d21593e225bfa4df"),
            "1" : ObjectId("617958a1d21593e225bfa4e0"),
            "2" : ObjectId("617958a1d21593e225bfa4e1"),
            "3" : ObjectId("617958a1d21593e225bfa4e2")
        },
        "value" : {
            "0" : 0.610814010052608,
            "1" : 0.51848041136916,
            "2" : 0.639634772233867,
            "3" : 0.975665115747611
        },
        "date" : {
            "0" : ISODate("2021-10-27T13:47:52.883Z"),
            "1" : ISODate("2021-10-27T19:47:52.883Z"),
            "2" : ISODate("2021-10-28T01:47:52.883Z"),
            "3" : ISODate("2021-10-28T07:47:52.883Z")
        }
    }
}/* 4 */
{
    "_id" : ObjectId("6179f61032850201c3dc46d0"),
    "control" : {
        "version" : 1,
        "min" : {
            "_id" : ObjectId("617958a1d21593e225bfa4d7"),
            "value" : 0.355601537158719,
            "date" : ISODate("2021-10-28T01:00:00.000Z")
        },
        "max" : {
            "_id" : ObjectId("617958a1d21593e225bfa4da"),
            "value" : 0.907835596255776,
            "date" : ISODate("2021-10-28T19:47:52.883Z")
        }
    },
    "meta" : {
        "y" : "a"
    },
    "data" : {
        "_id" : {
            "0" : ObjectId("617958a1d21593e225bfa4d7"),
            "1" : ObjectId("617958a1d21593e225bfa4d8"),
            "2" : ObjectId("617958a1d21593e225bfa4d9"),
            "3" : ObjectId("617958a1d21593e225bfa4da")
        },
        "value" : {
            "0" : 0.907835596255776,
            "1" : 0.478167641630244,
            "2" : 0.734791557369546,
            "3" : 0.355601537158719
        },
        "date" : {
            "0" : ISODate("2021-10-28T01:47:52.883Z"),
            "1" : ISODate("2021-10-28T07:47:52.883Z"),
            "2" : ISODate("2021-10-28T13:47:52.883Z"),
            "3" : ISODate("2021-10-28T19:47:52.883Z")
        }
    }
}/* 5 */
{
    "_id" : ObjectId("617a9ed032850201c3dc46d3"),
    "control" : {
        "version" : 1,
        "min" : {
            "_id" : ObjectId("617958a1d21593e225bfa4e3"),
            "value" : 0.651105330652535,
            "date" : ISODate("2021-10-28T13:00:00.000Z")
        },
        "max" : {
            "_id" : ObjectId("617958a1d21593e225bfa4e6"),
            "value" : 0.941601991088971,
            "date" : ISODate("2021-10-29T07:47:52.883Z")
        }
    },
    "meta" : {
        "y" : "a"
    },
    "data" : {
        "_id" : {
            "0" : ObjectId("617958a1d21593e225bfa4e3"),
            "1" : ObjectId("617958a1d21593e225bfa4e4"),
            "2" : ObjectId("617958a1d21593e225bfa4e5"),
            "3" : ObjectId("617958a1d21593e225bfa4e6")
        },
        "value" : {
            "0" : 0.941601991088971,
            "1" : 0.651105330652535,
            "2" : 0.725467962749386,
            "3" : 0.694043071407878
        },
        "date" : {
            "0" : ISODate("2021-10-28T13:47:52.883Z"),
            "1" : ISODate("2021-10-28T19:47:52.883Z"),
            "2" : ISODate("2021-10-29T01:47:52.883Z"),
            "3" : ISODate("2021-10-29T07:47:52.883Z")
        }
    }
}/* 6 */
{
    "_id" : ObjectId("617b479032850201c3dc46d1"),
    "control" : {
        "version" : 1,
        "min" : {
            "_id" : ObjectId("617958a1d21593e225bfa4db"),
            "value" : 0.0392149962590065,
            "date" : ISODate("2021-10-29T01:00:00.000Z")
        },
        "max" : {
            "_id" : ObjectId("617958a1d21593e225bfa4de"),
            "value" : 0.912639155007598,
            "date" : ISODate("2021-10-29T19:47:52.883Z")
        }
    },
    "meta" : {
        "y" : "a"
    },
    "data" : {
        "_id" : {
            "0" : ObjectId("617958a1d21593e225bfa4db"),
            "1" : ObjectId("617958a1d21593e225bfa4dc"),
            "2" : ObjectId("617958a1d21593e225bfa4dd"),
            "3" : ObjectId("617958a1d21593e225bfa4de")
        },
        "value" : {
            "0" : 0.0392149962590065,
            "1" : 0.715255277128178,
            "2" : 0.912639155007598,
            "3" : 0.877510759928182
        },
        "date" : {
            "0" : ISODate("2021-10-29T01:47:52.883Z"),
            "1" : ISODate("2021-10-29T07:47:52.883Z"),
            "2" : ISODate("2021-10-29T13:47:52.883Z"),
            "3" : ISODate("2021-10-29T19:47:52.883Z")
        }
    }
}/* 7 */
{
    "_id" : ObjectId("617bf05032850201c3dc46d4"),
    "control" : {
        "version" : 1,
        "min" : {
            "_id" : ObjectId("617958a1d21593e225bfa4e7"),
            "value" : 0.237759276862083,
            "date" : ISODate("2021-10-29T13:00:00.000Z")
        },
        "max" : {
            "_id" : ObjectId("617958a1d21593e225bfa4e8"),
            "value" : 0.815200486877654,
            "date" : ISODate("2021-10-29T19:47:52.883Z")
        }
    },
    "meta" : {
        "y" : "a"
    },
    "data" : {
        "_id" : {
            "0" : ObjectId("617958a1d21593e225bfa4e7"),
            "1" : ObjectId("617958a1d21593e225bfa4e8")
        },
        "value" : {
            "0" : 0.237759276862083,
            "1" : 0.815200486877654
        },
        "date" : {
            "0" : ISODate("2021-10-29T13:47:52.883Z"),
            "1" : ISODate("2021-10-29T19:47:52.883Z")
        }
    }
} 

4. Notice that the second record ("61794d5032850201c3dc46ce") has only one data point that could be grouped together with the 3rd record ("61794d5032850201c3dc46d2").

 

Participants:

 Description   

I was testing the new time series collections and I stumbled upon the following scenario: inserting many records at once that are not strictly ordered by date creates some weird results.



 Comments   
Comment by Edwin Zhou [ 04/Nov/21 ]

Hi razvan.virjoghe@gmail.com,

Thanks for your report and reproduction of this behavior. We believe this can be deduplicated against SERVER-60894, which describes that timeseries buckets can close when inserting out-of-order measurements or on restart. As you mention, this reduces our ability to optimize these measurements.

I'll close this ticket as a duplicate of SERVER-60894.

Best,
Edwin

Comment by razvan virjgohe [ 27/Oct/21 ]

If the array is completely unordered then the results can be even more dramatic leading to a 1:1 ration between datapoints and records in the buckets collection, meaning no optimization whatsoever.

Generated at Thu Feb 08 05:51:22 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.