Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-12325

[Spark] Add spark.jars.packages config to all Python config examples

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Spark Connector
    • Labels:
    • Environment:
      python

      Description

      On the spark connector python guide pages, it describes how to create spark session the documentation reads:

      from pyspark.sql import SparkSession
       
      my_spark = SparkSession \
       
      .builder \
       
      .appName("myApp") \
       
      .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/test.coll") \
       
      .config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.coll") \
       
      .getOrCreate()
      
      

      the snippet misses one more config param, which is the mongo spark connector  package

      the code should look like this

      from pyspark.sql import SparkSession
       
      my_spark = SparkSession \
       
      .builder \
       
      .appName("myApp") \
       
      .config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.4.0") \
       
      .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/test.coll") \
       
      .config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.coll") \ 
       
      .getOrCreate()
      
      

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            mhameed Mohammed Hameed
            Participants:
            Last commenter:
            Nathan Leniz Nathan Leniz
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Days since reply:
              4 weeks, 1 day ago
              Date of 1st Reply: