Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-413

Provide support for Spark >=v3.2.4

XMLWordPrintableJSON

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 10.3.0
    • Affects Version/s: None
    • Component/s: API, build
    • Labels:
      None
    • Needed
    • Hide

      1. What would you like to communicate to the user about this feature?

      We now support Spark 3.1 - 3.5
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      n/a

      3. Which versions of the driver/connector does this apply to?
      10.3.0

      Show
      1. What would you like to communicate to the user about this feature? We now support Spark 3.1 - 3.5 2. Would you like the user to see examples of the syntax and/or executable code and its output? n/a 3. Which versions of the driver/connector does this apply to? 10.3.0

      Spark connector v10.0.2 supports Spark server up to v3.2.4. More recent Spark server version (like 3.4.1 and 3.5.0) introduce breaking changes that need support in the Spark connector.

      [A community user reported this error when using spark 3.4.1 with Spark Connector 10.2](https://www.mongodb.com/community/forums/t/getting-java-lang-nosuchmethoderror-when-trying-to-write-to-a-collection-via-spark-mongo-connector/237021/2)

      The error message the Spark job is failing due to issues related to the InternalRowToRowFunction and RowToBsonDocumentConverter classes from the MongoDB Spark Connector.

      23/07/27 13:51:03 INFO DAGScheduler: ResultStage 1 (save at Main.java:25) failed in 12.753 s due to Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4) (10.0.7.3 executor 1): java.lang.NoSuchMethodError: 'scala.collection.immutable.Seq org.apache.spark.sql.types.StructType.toAttributes()' at com.mongodb.spark.sql.connector.schema.InternalRowToRowFunction.<init>(InternalRowToRowFunction.java:46) 

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            prakul.agarwal@mongodb.com Prakul Agarwal
            Votes:
            10 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: