-
Type: Task
-
Resolution: Works as Designed
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: None
-
Labels:None
I used to configure the uri's at the spark session level, e.g.
my_spark = SparkSession \
.builder \
.appName("myApp") \
.config("spark.mongodb.input.uri", "mongodb+[srv://tempuser:temppassword@msft-workshop-tp2me.azure.mongodb.net/test|srv://msft-workshop-tp2me.azure.mongodb.net/test]") \
.config("spark.mongodb.output.uri", "mongodb+[srv://tempuser:temppassword@msft-workshop-tp2me.azure.mongodb.net/test|srv://msft-workshop-tp2me.azure.mongodb.net/test]") \
.getOrCreate()
Then I'd run a simple test to load data into mdb using code like the following:
people = my_spark.createDataFrame([("Bilbo Baggins", 50), ("Gandalf", 1000), ("Thorin", 195), ("Balin", 178), ("Kili", 77),
("Dwalin", 169), ("Oin", 167), ("Gloin", 158), ("Fili", 82), ("Bombur", None)], ["name", "age"])
people.write.format("com.mongodb.spark.sql.DefaultSource").mode("append").option("database","spark").option("collection", "contacts").save()
I'm getting the following error:
IllegalArgumentException: "requirement failed: Missing 'uri' property from options"
This used to work with the 2.3 version of the spark connector.
In order to work around this, I now have to pass in the uri with every call which is an eye sore, e.g.
people.write.format("com.mongodb.spark.sql.DefaultSource").mode("append").option("database","spark").option("collection", "contacts").option("uri", "mongodb+[srv://tempuser]:temppassword@msft-azure-tp2me.azure.mongodb.net/test").save()
How do I set the uri globally at the spark session level?