-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Unknown
-
Affects Version/s: None
-
Component/s: None
-
None
-
None
-
None
-
None
-
None
-
None
When using new 10.0.0 connector with kotlin, arrays are converted into ArrayList (in BsonDocumentToRowConverter) and resulting into this error :
Caused by: java.lang.RuntimeException: Error while encoding: java.lang.RuntimeException: java.util.ArrayList is not a valid external type for schema of array<double>
Versions used
I am using spark-core, spark-sql, spark-catalyst in version 3.2.1 and mongo-spark-connector in 10.0.0
How to reproduce
Insert this document in mongo
{
"coordinates": {
"type": "Point",
"coordinates": [
0.0002895,
50.4227494
]
}
}
Then try to read it in spark with mongo connector :
val spark = SparkSession.builder()
.master("local")
.appName("Test")
.config("spark.mongodb.read.connection.uri", "localhost:27017/test")
.orCreate
val rdd = spark.read().format("mongodb").load()
rdd.printSchema()
rdd.show()
Debugging the code lead me to think that the problem is coming from ArrayList used in convertToArray function in BsonDocumentToRowConverter. Java ArrayList is not mapped to Kotlin List from what I understand.