Uploaded image for project: 'Core Server'
  1. Core Server
  2. SERVER-59505

Time-series query on mixed, nested measurements can miss some events

    XMLWordPrintable

    Details

    • Operating System:
      ALL
    • Backport Requested:
      v5.1, v5.0
    • Sprint:
      QO 2021-09-06, QO 2021-10-04, QO 2021-10-18, QO 2021-11-01

      Description

      Normally when a measurement field 'x' contains an array, the 'control.min.x' field on the bucket contains the min for each position in the array.

      > db.events.find()
      { x: [0, 2] }
      { x: [3, 1] }
       
      > db.system.buckets.events.find()
      {
        control: {
          min: {x: [0, 1]},
          max: {x: [3, 2]},
        },
        ...
      }
      

      But if 'x' also contains non-arrays, then 'control.min.x' or 'control.max.x' will be a non-array:

      > db.events.find()
      { x: [0, 2] }
      { x: [3, 1] }
      { x: 10 }
       
      > db.system.buckets.events.find()
      {
        control: {
          min: {x: 10},
          max: {x: [3, 2]},
        },
        ...
      }
      

      The predicate pushdowns don't account for this, so "multikey" queries can incorrectly exclude this bucket:

      > db.events.find({ x: {$lt: 5} })
      (no results)
       
      // because internally:
      > db.system.buckets.events.find({$expr: {$lt: ["$control.min.x", 5]}})
      

      A similar thing can happen if 'x' is a mixture of objects and non-objects:

      > db.events.find()
      { "time" : ..., "x" : ISODate("1970-01-01T00:00:00Z"), "_id" : ... }
      { "time" : ..., "x" : { "y" : ISODate("2021-01-02T11:59:59Z") }, "_id" : ...) }
       
      > db.events.find({ 'x.y': {$gt: ISODate('2000-01-01')} })
      (no results)
      

      This happens because although 'control.max.x' is the max of 'x', 'control.max.x.y' is not the max of 'x.y'. ('control.max.x.y' is 'missing', but 'missing' < ISODate.)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              samuel.mercier Sam Mercier
              Reporter:
              david.percy David Percy
              Participants:
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

                Dates

                Created:
                Updated: