Details
-
Type:
Task
-
Status: Closed
-
Priority:
Major - P3
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: Spark Connector
-
Labels:
-
Environment:python
-
Last comment by Customer:true
-
Epic Link:
Description
On the spark connector python guide pages, it describes how to create spark session the documentation reads:
from pyspark.sql import SparkSession |
|
|
my_spark = SparkSession \ |
|
|
.builder \
|
|
|
.appName("myApp") \ |
|
|
.config("spark.mongodb.input.uri", "mongodb://127.0.0.1/test.coll") \ |
|
|
.config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.coll") \ |
|
|
.getOrCreate()
|
|
the snippet misses one more config param, which is the mongo spark connector package
the code should look like this
from pyspark.sql import SparkSession |
|
|
my_spark = SparkSession \ |
|
|
.builder \
|
|
|
.appName("myApp") \ |
|
|
.config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.4.0") \ |
|
|
.config("spark.mongodb.input.uri", "mongodb://127.0.0.1/test.coll") \ |
|
|
.config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.coll") \ |
|
|
.getOrCreate()
|
|