Hi Ross,
I actually just found out the problem.
Even if I put replaceDocument to writeConfig:
val writeConfig = WriteConfig(Map("uri" -> s"mongodb://$host/$database.$table", "replaceDocument" -> "false"))
If I use:
MongoSpark.save(data.write.mode("append"), writeConfig)
It will not work, because this function call the dataFrameWriter from Spark and doesn't take account of the mongodb writeConfig options. Instead I should use:
MongoSpark.save(data, writeConfig)
This will call the correct MongoSpark writer function.
I think this point should be noted in the document because it is indeed confusing.
Regards,
Thang
https://groups.google.com/d/msg/mongodb-user/eF-qdpYbFS0/KNqDB2vJAwAJ
- is depended on by
-
SPARK-181 Spark Connector 2.3.0
- Closed