Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-4691

non-UTC timezones break DATETIME_CLAMP/DATETIME_AUTO

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 4.9
    • Affects Version/s: None
    • Component/s: None
    • None
    • Python Drivers
    • Not Needed
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      Discovered while working on PYTHON-4663, DATETIME_CLAMP and DATETIME_AUTO break on non-UTC timezones:

      from bson import encode, decode, CodecOptions, DatetimeConversion
      from bson.tz_util import utc, FixedOffset
      import datetime
      
      tz = FixedOffset(60, "Custom")
      opts = CodecOptions(datetime_conversion=DatetimeConversion.DATETIME_CLAMP, tz_aware=True, tzinfo=tz)
      now = datetime.datetime.now(tz)
      doc = {"d": now}
      print(f'doc: {doc}')
      print(f'decoded as naive: {decode(encode(doc))}' )
      decode(encode(doc), opts)  # Fails
      

      Output:

      doc: {'d': datetime.datetime(2024, 8, 23, 0, 20, 28, 759709, tzinfo=FixedOffset(datetime.timedelta(seconds=3600), 'Custom'))}
      decoded as naive: {'d': datetime.datetime(2024, 8, 22, 23, 20, 28, 759000)}
      Traceback (most recent call last):
        File "/Users/shane/git/mongo-python-driver/repro.py", line 11, in <module>
          decode(encode(doc), opts)
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 1078, in decode
          return cast("Union[dict[str, Any], _DocumentType]", _bson_to_dict(data, opts))
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 618, in _bson_to_dict
          raise InvalidBSON(str(exc_value)).with_traceback(exc_tb) from None
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 612, in _bson_to_dict
          return cast("_DocumentType", _elements_to_dict(data, view, 4, end, opts))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 596, in _elements_to_dict
          key, value, position = _element_to_dict(
                                 ^^^^^^^^^^^^^^^^^
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 551, in _element_to_dict
          value, position = _ELEMENT_GETTER[element_type](
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 422, in _get_date
          return _millis_to_datetime(_UNPACK_LONG_FROM(data, position)[0], opts), position + 8
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/shane/git/mongo-python-driver/bson/datetime_ms.py", line 141, in _millis_to_datetime
          millis = max(_min_datetime_ms(tz), min(millis, _max_datetime_ms(tz)))
                       ^^^^^^^^^^^^^^^^^^^^
        File "/Users/shane/git/mongo-python-driver/bson/datetime_ms.py", line 122, in _min_datetime_ms
          return _datetime_to_millis(datetime.datetime.min.replace(tzinfo=tz))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/shane/git/mongo-python-driver/bson/datetime_ms.py", line 170, in _datetime_to_millis
          dtm = dtm - dtm.utcoffset()  # type: ignore
                ~~~~^~~~~~~~~~~~~~~~~
      bson.errors.InvalidBSON: date value out of range
      

      One problem is that _min_datetime_ms/_max_datetime_ms are not accurate with non-utc timezones:

      @functools.lru_cache(maxsize=None)
      def _min_datetime_ms(tz: datetime.timezone = datetime.timezone.utc) -> int:
          return _datetime_to_millis(datetime.datetime.min.replace(tzinfo=tz))
      
      
      @functools.lru_cache(maxsize=None)
      def _max_datetime_ms(tz: datetime.timezone = datetime.timezone.utc) -> int:
          return _datetime_to_millis(datetime.datetime.max.replace(tzinfo=tz))
      

      Notice the difference here:

      >>> tz = FixedOffset(60, "Custom")
      >>> datetime.datetime.min.replace(tzinfo=tz)
      datetime.datetime(1, 1, 1, 0, 0, tzinfo=FixedOffset(datetime.timedelta(seconds=3600), 'Custom'))
      >>> datetime.datetime.min.replace(tzinfo=utc).astimezone(tz)
      datetime.datetime(1, 1, 1, 1, 0, tzinfo=FixedOffset(datetime.timedelta(seconds=3600), 'Custom'))
      >>> datetime.datetime.max.replace(tzinfo=utc)
      datetime.datetime(9999, 12, 31, 23, 59, 59, 999999, tzinfo=FixedOffset(datetime.timedelta(0), 'UTC'))
      >>> datetime.datetime.max.replace(tzinfo=utc).astimezone(tz)
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
      OverflowError: date value out of range
      

      The OverflowError on datetime.max highlights an important point: The max (or min) datetime in one timezone can be outside the representable range of another timezone. To calculate the actual bounds we need to adjust the max/min by the timezone's utcoffset():

      >>> max_utc = datetime.datetime.max.replace(tzinfo=utc)
      >>> max_tz = (max_utc - tz.utcoffset(max_utc)).astimezone(tz)
      >>> max_tz
      datetime.datetime(9999, 12, 31, 23, 59, 59, 999999, tzinfo=FixedOffset(datetime.timedelta(seconds=3600), 'Custom'))
      >>> max_tz.astimezone(utc)
      datetime.datetime(9999, 12, 31, 22, 59, 59, 999999, tzinfo=FixedOffset(datetime.timedelta(0), 'UTC'))
      

            Assignee:
            shane.harvey@mongodb.com Shane Harvey
            Reporter:
            shane.harvey@mongodb.com Shane Harvey
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: