Uploaded image for project: 'Python Integrations'
  1. Python Integrations
  2. INTPYTHON-389

Avoid Type Mismatch Conversion to NaN

    • Type: Icon: Bug Bug
    • Resolution: Unresolved
    • Priority: Icon: Unknown Unknown
    • pymongoarrow-1.7
    • Affects Version/s: None
    • Component/s: pymongoarrow
    • None
    • Python Drivers
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      Via https://github.com/mongodb-labs/mongo-arrow/issues/246

       

      When using `find` methods with a defined schema, values in MongoDB documents that don’t match the specified schema types (e.g., strings in a field expected to be int64) are silently converted to NaN in the results. This behavior can lead to unintended data loss and inconsistency, as there’s no indication of the type mismatch or a warning about the data conversion.

      Ideally, the function should raise an error indicating that a type mismatch has been detected in the value field, rather than converting to NaN. This would make the `find` method more robust and prevent data integrity issues due to unintentional type mismatches.

      Example:
      ```
      from pymongoarrow.monkey import patch_all
      patch_all()
      from pymongo import MongoClient
      from pymongoarrow.api import Schema
      import pyarrow as pa

      coll = MongoClient().test.test
      coll.insert_many([\{"value": 1}, \{"value": "1"}, \{"value": 1.1}, \{"value": None}, \{"value": False}, {}])

      schema = {"value": pa.int64()}
      projection = {"_id": 0, "value": 1}
      df = coll.find_pandas_all(query={}, schema=Schema(schema), projection=projection)
      print(df)
      ```

      Output:
      ```
         value
      0    1.0
      1    NaN
      2    1.0
      3    NaN
      4    0.0
      5    NaN
      ```

            Assignee:
            Unassigned Unassigned
            Reporter:
            alex.clark@mongodb.com Alex Clark
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: