Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-238

sparksession uri parameter no longer working for 2.4.0 spark connector

    • Type: Icon: Task Task
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None

      I used to configure the uri's at the spark session level, e.g.

       
      my_spark = SparkSession \
          .builder \
          .appName("myApp") \
          .config("spark.mongodb.input.uri", "mongodb+[srv://tempuser:temppassword@msft-workshop-tp2me.azure.mongodb.net/test|srv://msft-workshop-tp2me.azure.mongodb.net/test]") \
          .config("spark.mongodb.output.uri", "mongodb+[srv://tempuser:temppassword@msft-workshop-tp2me.azure.mongodb.net/test|srv://msft-workshop-tp2me.azure.mongodb.net/test]") \
          .getOrCreate()
       
      Then I'd run a simple test to load data into mdb using code like the following:
       
      people = my_spark.createDataFrame([("Bilbo Baggins",  50), ("Gandalf", 1000), ("Thorin", 195), ("Balin", 178), ("Kili", 77),
         ("Dwalin", 169), ("Oin", 167), ("Gloin", 158), ("Fili", 82), ("Bombur", None)], ["name", "age"])
      people.write.format("com.mongodb.spark.sql.DefaultSource").mode("append").option("database","spark").option("collection", "contacts").save()
       
      I'm getting the following error:
       
      IllegalArgumentException: "requirement failed: Missing 'uri' property from options"
       
      This used to work with the 2.3 version of the spark connector.
       
      In order to work around this, I now have to pass in the uri with every call which is an eye sore, e.g. 
       
      people.write.format("com.mongodb.spark.sql.DefaultSource").mode("append").option("database","spark").option("collection", "contacts").option("uri", "mongodb+[srv://tempuser]:temppassword@msft-azure-tp2me.azure.mongodb.net/test").save()
       
      How do I set the uri globally at the spark session level?
       
       
       

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            eugene.kang@mongodb.com Eugene Kang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: