-
Type: Bug
-
Resolution: Fixed
-
Priority: Unknown
-
Affects Version/s: None
-
Component/s: None
-
None
-
Python Drivers
-
Not Needed
-
Discovered while working on PYTHON-4663, DATETIME_CLAMP and DATETIME_AUTO break on non-UTC timezones:
from bson import encode, decode, CodecOptions, DatetimeConversion from bson.tz_util import utc, FixedOffset import datetime tz = FixedOffset(60, "Custom") opts = CodecOptions(datetime_conversion=DatetimeConversion.DATETIME_CLAMP, tz_aware=True, tzinfo=tz) now = datetime.datetime.now(tz) doc = {"d": now} print(f'doc: {doc}') print(f'decoded as naive: {decode(encode(doc))}' ) decode(encode(doc), opts) # Fails
Output:
doc: {'d': datetime.datetime(2024, 8, 23, 0, 20, 28, 759709, tzinfo=FixedOffset(datetime.timedelta(seconds=3600), 'Custom'))} decoded as naive: {'d': datetime.datetime(2024, 8, 22, 23, 20, 28, 759000)} Traceback (most recent call last): File "/Users/shane/git/mongo-python-driver/repro.py", line 11, in <module> decode(encode(doc), opts) File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 1078, in decode return cast("Union[dict[str, Any], _DocumentType]", _bson_to_dict(data, opts)) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 618, in _bson_to_dict raise InvalidBSON(str(exc_value)).with_traceback(exc_tb) from None File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 612, in _bson_to_dict return cast("_DocumentType", _elements_to_dict(data, view, 4, end, opts)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 596, in _elements_to_dict key, value, position = _element_to_dict( ^^^^^^^^^^^^^^^^^ File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 551, in _element_to_dict value, position = _ELEMENT_GETTER[element_type]( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shane/git/mongo-python-driver/bson/__init__.py", line 422, in _get_date return _millis_to_datetime(_UNPACK_LONG_FROM(data, position)[0], opts), position + 8 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shane/git/mongo-python-driver/bson/datetime_ms.py", line 141, in _millis_to_datetime millis = max(_min_datetime_ms(tz), min(millis, _max_datetime_ms(tz))) ^^^^^^^^^^^^^^^^^^^^ File "/Users/shane/git/mongo-python-driver/bson/datetime_ms.py", line 122, in _min_datetime_ms return _datetime_to_millis(datetime.datetime.min.replace(tzinfo=tz)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/shane/git/mongo-python-driver/bson/datetime_ms.py", line 170, in _datetime_to_millis dtm = dtm - dtm.utcoffset() # type: ignore ~~~~^~~~~~~~~~~~~~~~~ bson.errors.InvalidBSON: date value out of range
One problem is that _min_datetime_ms/_max_datetime_ms are not accurate with non-utc timezones:
@functools.lru_cache(maxsize=None) def _min_datetime_ms(tz: datetime.timezone = datetime.timezone.utc) -> int: return _datetime_to_millis(datetime.datetime.min.replace(tzinfo=tz)) @functools.lru_cache(maxsize=None) def _max_datetime_ms(tz: datetime.timezone = datetime.timezone.utc) -> int: return _datetime_to_millis(datetime.datetime.max.replace(tzinfo=tz))
Notice the difference here:
>>> tz = FixedOffset(60, "Custom") >>> datetime.datetime.min.replace(tzinfo=tz) datetime.datetime(1, 1, 1, 0, 0, tzinfo=FixedOffset(datetime.timedelta(seconds=3600), 'Custom')) >>> datetime.datetime.min.replace(tzinfo=utc).astimezone(tz) datetime.datetime(1, 1, 1, 1, 0, tzinfo=FixedOffset(datetime.timedelta(seconds=3600), 'Custom')) >>> datetime.datetime.max.replace(tzinfo=utc) datetime.datetime(9999, 12, 31, 23, 59, 59, 999999, tzinfo=FixedOffset(datetime.timedelta(0), 'UTC')) >>> datetime.datetime.max.replace(tzinfo=utc).astimezone(tz) Traceback (most recent call last): File "<stdin>", line 1, in <module> OverflowError: date value out of range
The OverflowError on datetime.max highlights an important point: The max (or min) datetime in one timezone can be outside the representable range of another timezone. To calculate the actual bounds we need to adjust the max/min by the timezone's utcoffset():
>>> max_utc = datetime.datetime.max.replace(tzinfo=utc) >>> max_tz = (max_utc - tz.utcoffset(max_utc)).astimezone(tz) >>> max_tz datetime.datetime(9999, 12, 31, 23, 59, 59, 999999, tzinfo=FixedOffset(datetime.timedelta(seconds=3600), 'Custom')) >>> max_tz.astimezone(utc) datetime.datetime(9999, 12, 31, 22, 59, 59, 999999, tzinfo=FixedOffset(datetime.timedelta(0), 'UTC'))
- related to
-
PYTHON-4663 dateutil timezones do not work with DATETIME_CLAMP/DATETIME_AUTO
- Closed