-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Unknown
-
Affects Version/s: None
-
Component/s: None
-
None
-
Needed
-
When i try to save a dataframe that contains a WrappedArray with a null value inside it to MongoDB, it throws an exception of type com.mongodb.spark.sql.connector.exceptions.DataException. The full error message is the following
Cannot cast [[WrappedArray(null)],63aacee6e3fe179cabd46a30] into a BsonValue. StructType(StructField(EnergyMarket,StructType(StructField(balancingRequirements,ArrayType(StringType,true),true)),true),StructField(_id,StringType,true)) has no matching BsonValue. Error: Cannot cast [WrappedArray(null)] into a BsonValue. StructType(StructField(balancingRequirements,ArrayType(StringType,true),true)) has no matching BsonValue. Error: Cannot cast WrappedArray(null) into a BsonValue. ArrayType(StringType,true) has no matching BsonValue. Error: Cannot cast null into a BsonValue. StringType has no matching BsonValue. Error: Value can not be null
The schema of the dataframe is
|-- EnergyMarket: struct (nullable = true) | |-- balancingRequirements: array (nullable = true) | | |-- element: string (containsNull = true) |-- _id: string (nullable = true)
A sample of the actual dataframe's data
+------------+--------------------+
|EnergyMarket| _id|
+------------+--------------------+
| {[null]}|63aacee6e3fe179ca...|
| {[test]}|63aacee6e3fe179ca...|
+------------+--------------------+
The same data in json format
[
{
"_id": {
"$oid": "63aacee6e3fe179cabd46a30"
},
"EnergyMarket": {
"balancingRequirements": [
null
]
}
},
{
"_id": {
"$oid": "63aacee6e3fe179cabd46a31"
},
"EnergyMarket": {
"balancingRequirements": [
"test"
]
}
}
]
Finally, I would like to mention that I am using the version 10.0.5 of Spark Connector. This problem did not exist with the previous version (3.0.1).