Uploaded image for project: 'Python Integrations'
  1. Python Integrations
  2. INTPYTHON-84

Add option to use pyarrow decimal128 type

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None

      In ARROW-15 we added support for treating BSON Decimal128 as string types.

      This allows for a clean mapping with no data loss, at the expense of slowness and the awkwardness of having to convert the returned string value to {Decimal}.

      We should look into also offering the ability to map BSON Decimal128 values to PyArrow decimal128 values.  Care must be taken to ensure there is no data loss, since the two data structures are not directly compatible.

      The BSON Decimal128 spec is defined here. It uses an IEEE 754-2008 decimal128 value.

      The arrow type is a fundamentally different type unrelated to IEEE 754-2008. The arrow type is expecting a fixed scale for the data type, where BSON decimal128 values have a precision of 34 digits and a variable scale. The arrow type is meant for a set of data that has a consistent representation, like dollar amounts.

       

       

            Assignee:
            Unassigned Unassigned
            Reporter:
            steve.silvester@mongodb.com Steve Silvester
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: