-
Type: Bug
-
Resolution: Fixed
-
Priority: Unknown
-
Affects Version/s: None
-
Component/s: None
When using new 10.0.0 connector with kotlin, arrays are converted into ArrayList (in BsonDocumentToRowConverter) and resulting into this error :
Caused by: java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: java.util.ArrayList is not a valid external type for schema of array<double>
Versions used
I am using spark-core, spark-sql, spark-catalyst in version 3.2.1 and mongo-spark-connector in 10.0.0
How to reproduce
Insert this document in mongo
{ "coordinates": { "type": "Point", "coordinates": [ 0.0002895, 50.4227494 ] } }
Then try to read it in spark with mongo connector :
val spark = SparkSession.builder() .master("local") .appName("Test") .config("spark.mongodb.read.connection.uri", "localhost:27017/test") .orCreate val rdd = spark.read().format("mongodb").load() rdd.printSchema() rdd.show()
Debugging the code lead me to think that the problem is coming from ArrayList used in convertToArray function in BsonDocumentToRowConverter. Java ArrayList is not mapped to Kotlin List from what I understand.