Uploaded image for project: 'Python Driver'
  1. Python Driver
  2. PYTHON-3846

Optimize BSON encoding of large ints

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 4.5
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None

      When encoding an int the C extensions first attempt to encode as an int32, then check for an overflow error, then attempt to encode as an int64:

          else if (PyLong_Check(value)) {
              const long long_value = PyLong_AsLong(value);
      
              const int int_value = (int)long_value;
              if (PyErr_Occurred() || long_value != int_value) { /* Overflow */
                  long long long_long_value;
                  PyErr_Clear();
                  long_long_value = PyLong_AsLongLong(value);
                  if (PyErr_Occurred()) {
                      /* Ignore error and give the fallback_encoder a chance. */
                      PyErr_Clear();
                  } else {
                      *(pymongo_buffer_get_buffer(buffer) + type_byte) = 0x12;
                      return buffer_write_int64(buffer, (int64_t)long_long_value);
                  }
              } else {
                  *(pymongo_buffer_get_buffer(buffer) + type_byte) = 0x10;
                  return buffer_write_int32(buffer, (int32_t)int_value);
              }
      

      https://github.com/mongodb/mongo-python-driver/blob/469e2e95f5eedbbb98d30d3309327c7bf19d0eb7/bson/_cbsonmodule.c#L1113-L1131

      It would be faster to encode to a int64 first and then write the value as an int32 or int64 based on the value, like we do in the equivalent Python code:

      def _encode_int(name: bytes, value: int, dummy0: Any, dummy1: Any) -> bytes:
          """Encode a python int."""
          if -2147483648 <= value <= 2147483647:
              return b"\x10" + name + _PACK_INT(value)
          else:
              try:
                  return b"\x12" + name + _PACK_LONG(value)
      

      https://github.com/mongodb/mongo-python-driver/blob/469e2e95f5eedbbb98d30d3309327c7bf19d0eb7/bson/__init__.py#L799-L805

            Assignee:
            shane.harvey@mongodb.com Shane Harvey
            Reporter:
            shane.harvey@mongodb.com Shane Harvey
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: