Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-180

MongoSpark.Save(DataFrameWriter, writeConfig) should respect replaceDocument.

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 2.3.0
    • Affects Version/s: None
    • Component/s: Writes
    • None

      Hi Ross,

      I actually just found out the problem.

      Even if I put replaceDocument to writeConfig:

      val writeConfig = WriteConfig(Map("uri" -> s"mongodb://$host/$database.$table", "replaceDocument" -> "false"))

      If I use:

      MongoSpark.save(data.write.mode("append"), writeConfig)

      It will not work, because this function call the dataFrameWriter from Spark and doesn't take account of the mongodb writeConfig options. Instead I should use:

      MongoSpark.save(data, writeConfig)

      This will call the correct MongoSpark writer function.

      I think this point should be noted in the document because it is indeed confusing.

      Regards,
      Thang

      https://groups.google.com/d/msg/mongodb-user/eF-qdpYbFS0/KNqDB2vJAwAJ

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            ross@mongodb.com Ross Lawley
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: