Core Server
  1. Core Server
  2. SERVER-1460

BSON should offer a high resolution 'UTC datetime' field type.

    Details

    • Backport:
      No
    • # Replies:
      8
    • Last comment by Customer:
      true

      Description

      The BSON spec defines a 'UTC datetime' field, specified as number of milliseconds since the epoch. However, milliseconds is often far too coarse grained for recording timestamps in high performance systems. It is also rather wasteful of the entire 64 bit range. A signed 64 bit integer can represent slightly less than 300 milliion years worth of milliseconds. That is a very long time.

      The BSON spec should offer a high resolution UTC datetime type, denominated either in microseconds (almost 300,000 years in an int64_t), or even nanoseconds (still almost 300 years). Perhaps both?

        Activity

        Hide
        Eliot Horowitz
        added a comment -

        We're trying to keep the types in the spec pretty low or all the drivers have to implement.

        In this case - since you can just store it in a regular long long field and know its a datetime from logic - is it really needed?

        Show
        Eliot Horowitz
        added a comment - We're trying to keep the types in the spec pretty low or all the drivers have to implement. In this case - since you can just store it in a regular long long field and know its a datetime from logic - is it really needed?
        Hide
        Andrew Morrow
        added a comment -

        Yes, I am currently storing high-res timestamps in a normal long long field. This works OK, but it isn't great, since things like BSONObj::toString and BSONObj::jsonString don't know that it is meant to be a timestamp, and just print it as a raw long integer value. That means that if I want to use b2json, say in a shell pipeline, then I would need to write some sort of post-processor that knows that certain things (based on key?) that look like 64 bit integers are really timestamps, and mangle them appropriately in order to get readable output. That is rather fragile.

        If BSONObj::jsonString was aware that it was a high-res timestamp, it could choose to represent it in some more useful format, and I wouldn't need to make a context dependent tool.

        Thinking about this some more, is there any way in the current BSONObj::toString or BSONObj::jsonString to specify how UTC datetime objects should be formatted, especially in the 'pretty' case of jsonString? Since I haven't used the UTC datetime field, due to its limited precision, I don't actually know how it gets printed.

        Anyway, I understand your point about keeping the type-space for BSON small, and that it is painful to add new types. However, I imagine I'm not the only one who will want something more fine-grained than milliseconds when representing timestamps.

        Show
        Andrew Morrow
        added a comment - Yes, I am currently storing high-res timestamps in a normal long long field. This works OK, but it isn't great, since things like BSONObj::toString and BSONObj::jsonString don't know that it is meant to be a timestamp, and just print it as a raw long integer value. That means that if I want to use b2json, say in a shell pipeline, then I would need to write some sort of post-processor that knows that certain things (based on key?) that look like 64 bit integers are really timestamps, and mangle them appropriately in order to get readable output. That is rather fragile. If BSONObj::jsonString was aware that it was a high-res timestamp, it could choose to represent it in some more useful format, and I wouldn't need to make a context dependent tool. Thinking about this some more, is there any way in the current BSONObj::toString or BSONObj::jsonString to specify how UTC datetime objects should be formatted, especially in the 'pretty' case of jsonString? Since I haven't used the UTC datetime field, due to its limited precision, I don't actually know how it gets printed. Anyway, I understand your point about keeping the type-space for BSON small, and that it is painful to add new types. However, I imagine I'm not the only one who will want something more fine-grained than milliseconds when representing timestamps.
        Hide
        Andrew Morrow
        added a comment -

        I should clarify: this is of particular interest w.r.t. to using BSON as a logging format, where the consumer is often a human trying to read log entries after conversion from BSON to JSON. Obviously, if the consumer is code, it is much easier for it to understand that certain fields, though represented as a raw int64, are really timestamps.

        Show
        Andrew Morrow
        added a comment - I should clarify: this is of particular interest w.r.t. to using BSON as a logging format, where the consumer is often a human trying to read log entries after conversion from BSON to JSON. Obviously, if the consumer is code, it is much easier for it to understand that certain fields, though represented as a raw int64, are really timestamps.
        Hide
        Brian Knox
        added a comment -

        Any plans for this feature? I ask because Rainer just added millisecond precision unix time to rsyslog, and it would be awesome to go straight into mongo with that information using the ommongodb output plugin. Thanks!

        Show
        Brian Knox
        added a comment - Any plans for this feature? I ask because Rainer just added millisecond precision unix time to rsyslog, and it would be awesome to go straight into mongo with that information using the ommongodb output plugin. Thanks!
        Hide
        Eliot Horowitz
        added a comment -

        @brian - datetime is already millisecond precision.

        this ticket is for micro (or nano)

        Show
        Eliot Horowitz
        added a comment - @brian - datetime is already millisecond precision. this ticket is for micro (or nano)
        Hide
        John Zwinck
        added a comment -

        I really want this feature, and I really want it to be with nanoseconds (else I'll be stuck where I am now, which is exactly where Andrew was stuck almost two years ago). Pretty please? This would be very valuable for people who write high-frequency events to MongoDB, regardless of syslog.

        This is literally the only extra field type I really wish BSON had, and I've used it for a while.

        Show
        John Zwinck
        added a comment - I really want this feature, and I really want it to be with nanoseconds (else I'll be stuck where I am now, which is exactly where Andrew was stuck almost two years ago). Pretty please? This would be very valuable for people who write high-frequency events to MongoDB, regardless of syslog. This is literally the only extra field type I really wish BSON had, and I've used it for a while.
        Hide
        Brian Knox
        added a comment -

        Oh! I should have read more carefully - well that's great news then.
        You have me the feature I wanted without having to do anything other than
        point out it already does it

        Brian

        Show
        Brian Knox
        added a comment - Oh! I should have read more carefully - well that's great news then. You have me the feature I wanted without having to do anything other than point out it already does it Brian
        Hide
        Victor Hooi
        added a comment -

        Hi,

        I can second that this feature would be awesome as well.

        We have high-precision events (nano-seconds) that we'd like to store in MongoDB, and have it be aware that they were datetimes as well.

        Any idea on whether this will ever be implemented?

        Cheers,
        Victor

        Show
        Victor Hooi
        added a comment - Hi, I can second that this feature would be awesome as well. We have high-precision events (nano-seconds) that we'd like to store in MongoDB, and have it be aware that they were datetimes as well. Any idea on whether this will ever be implemented? Cheers, Victor

          People

          • Votes:
            3 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Days since reply:
              1 year, 3 weeks, 5 days ago
              Date of 1st Reply: