Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-396

spark.mongodb.write.convertJson automatically converts data-types, without taking into account the data-type set in Spark

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 10.1.0, 10.1.1
    • Component/s: None
    • None
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      One of the recently added features (the spark.mongodb.write.convertJson option) handles the Bson specific data-types, like ObjectId, properly. But, it automatically converts other fields of other types as well, which is not the desired behaviour in every case.

      For example, in our use-case the product code field almost always is an integer value, and we store it as string (the type handling is done in Spark), the aforementioned feature automatically converts this field to integer type and it doesn’t offer any other option to override/affect this behaviour.

      This was not the case with the v3 of this connector, and it is blocking our organisation from using the newest version.

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            xhemilgota@odysseyprime.nl Xhemil Gota
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: