h3 What did I use

• Databricks Runtime Version 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)
  • org.mongodb.spark:mongo-spark-connector:10.0.1
• MongoDB 5.0

h3 What did I do

data = [
  ["tmp_user_1", [{"attr_1": "aaa"}, {"attr_2": "bbb"}]],
  ["tmp_user_2", []],
  ["tmp_user_3", None],
] 
columns = ["username", "attributes"] 

df = spark.createDataFrame(data, columns)

(
  df
  .write
  .format("mongodb")
  .option("database", database)
  .option("collection", collection)
  .option("connection.uri", connection_uri)
  .mode("overwrite")
  .save()
)

h3 What did I get

the data can't be written to mongodb

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 25.0 failed 1 times, most recent failure: Lost task 2.0 in stage 25.0 (TID 90) (ip-10-172-164-192.us-west-2.compute.internal executor driver): com.mongodb.spark.sql.connector.exceptions.DataException: Cannot cast [tmp_user_1,WrappedArray(Map(attr_1 -> aaa), Map(attr_2 -> bbb))] into a BsonValue. StructType(StructField(username,StringType,true),StructField(attributes,ArrayType(MapType(StringType,StringType,true),true),true)) has no matching BsonValue. Error: Cannot cast [Map(attr_1 -> aaa), Map(attr_2 -> bbb)] into a BsonValue. ArrayType(MapType(StringType,StringType,true),true) has no matching BsonValue. Error: Cannot cast Map(attr_1 -> aaa) into a BsonValue. MapType(StringType,StringType,true) has no matching BsonValue. Error: scala.collection.immutable.Map$Map1 cannot be cast to java.util.Map

What do I expect

The df can be written to mongodb