Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-364

Improve Spark Connector "overwrite" Write Mode

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Writes
    • Labels:
      None
    • Hide

      1. What would you like to communicate to the user about this feature?
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?

      Hi Team,

      Currently, the “overwrite” write mode drops the target collection and then proceeds to insert data into the collection.

      The problem is that when it recreates the collection it does so with default options and that creates problems for the following example scenarios:

      • Sharded Collections: as the new collection is not sharded anymore.
      • Collections with non-default collations
      • Time-series collections

      There may be others but these are the ones I could come up with. The intention behind this improvement request is that we add the appropriate functionality so that the "overwrite" mode recreates the collection with the same exact options as it was created.

      Otherwise this mode is rendered useless for several use cases.

      Regards
      Diego

            Assignee:
            Unassigned Unassigned
            Reporter:
            diego.rodriguez@mongodb.com Diego Rodriguez (Inactive)
            Votes:
            9 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: