Details
-
Task
-
Status: Closed
-
Major - P3
-
Resolution: Fixed
-
None
-
None
-
python
-
true
Description
On the spark connector python guide pages, it describes how to create spark session the documentation reads:
from pyspark.sql import SparkSession |
|
|
my_spark = SparkSession \ |
|
|
.builder \
|
|
|
.appName("myApp") \ |
|
|
.config("spark.mongodb.input.uri", "mongodb://127.0.0.1/test.coll") \ |
|
|
.config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.coll") \ |
|
|
.getOrCreate()
|
|
the snippet misses one more config param, which is the mongo spark connector package
the code should look like this
from pyspark.sql import SparkSession |
|
|
my_spark = SparkSession \ |
|
|
.builder \
|
|
|
.appName("myApp") \ |
|
|
.config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector_2.11:2.4.0") \ |
|
|
.config("spark.mongodb.input.uri", "mongodb://127.0.0.1/test.coll") \ |
|
|
.config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.coll") \ |
|
|
.getOrCreate()
|
|