Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-342

Error with kotlin when using arrays

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 10.0.1
    • Affects Version/s: None
    • Component/s: None
    • Labels:

      When using new 10.0.0 connector with kotlin, arrays are converted into ArrayList (in BsonDocumentToRowConverter) and resulting into this error : 

      Caused by: java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: java.util.ArrayList is not a valid external type for schema of array<double>

      Versions used

      I am using spark-core, spark-sql, spark-catalyst in version 3.2.1 and mongo-spark-connector in 10.0.0

      How to reproduce
      Insert this document in mongo 

      {
        "coordinates": {
          "type": "Point",
          "coordinates": [
            0.0002895,
            50.4227494
          ]
        }
      } 

      Then try to read it in spark with mongo connector : 

      val spark = SparkSession.builder()
          .master("local")
          .appName("Test")
          .config("spark.mongodb.read.connection.uri", "localhost:27017/test")
          .orCreate
      
      val rdd = spark.read().format("mongodb").load()
      rdd.printSchema()
      rdd.show() 

      Debugging the code lead me to think that the problem is coming from ArrayList used in convertToArray function in BsonDocumentToRowConverter. Java ArrayList is not mapped to Kotlin List from what I understand. 

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            julien.lerebours@enovea.net Julien LEREBOURS
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: