Possible Incompatibility in latest version of Spark - 3.4.1

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Works as Designed
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • None
    • Needed
    • Hide

      1. What would you like to communicate to the user about this feature?

      Upper Version limitation on the version of Spark supported
      2. Would you like the user to see examples of the syntax and/or executable code and its output?
      3. Which versions of the driver/connector does this apply to?

      Show
      1. What would you like to communicate to the user about this feature? Upper Version limitation on the version of Spark supported 2. Would you like the user to see examples of the syntax and/or executable code and its output? 3. Which versions of the driver/connector does this apply to?
    • None
    • None
    • None
    • None
    • None
    • None

       

      A community user reported this error when using spark 3.4.1 with Spark Connector 10.2

      https://www.mongodb.com/community/forums/t/getting-java-lang-nosuchmethoderror-when-trying-to-write-to-a-collection-via-spark-mongo-connector/237021/2

       

      The error message  the Spark job is failing due to issues related to the InternalRowToRowFunction and RowToBsonDocumentConverter classes from the MongoDB Spark Connector.

       

      23/07/27 13:51:03 INFO DAGScheduler: ResultStage 1 (save at Main.java:25) failed in 12.753 s due to Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 4) (10.0.7.3 executor 1): java.lang.NoSuchMethodError: 'scala.collection.immutable.Seq org.apache.spark.sql.types.StructType.toAttributes()' at com.mongodb.spark.sql.connector.schema.InternalRowToRowFunction.<init>(InternalRowToRowFunction.java:46) 

              Assignee:
              Unassigned
              Reporter:
              Prakul Agarwal
              None
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: