Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-200

mongo spark connector failed to load mongodb collection

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 2.1.4, 2.2.5, 2.3.1
    • Affects Version/s: 2.1.3, 2.2.4, 2.3.0
    • Component/s: Configuration
    • Labels:
      None
    • Environment:
      Centos

       
      Hello Team !
      I'm using zeppelin Version 0.8.0 ,mongodb 4.0, spark 2.2.0, mongospark connector 2.2.4, mongo java driver 3.8.
      I'm trying to load collection from mongodb for visualisation tasks using zeppelin
       
       
      import com.mongodb.spark.MongoSpark
      import com.mongodb.spark.config.{ReadConfig, WriteConfig}
      import com.mongodb.spark.sql._
      import org.apache.spark.sql.functions._
      import org.bson.Document
      import collection.JavaConverters._
      import org.apache.zeppelin.display.angular.paragraphscope._
      import AngularElem._
      val readConfig = ReadConfig(Map("uri" > "mongodb://127.0.0.1:27017/", "database" -> "test","collection" -> "Collection_f","allowDiskUse">"True"))
      val zipDf = spark.sparkSession.read.mongo(readConfig).toDF()
       
      it gives this error
       
      org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 2, localhost, executor driver): com.mongodb.MongoCommandException: Command failed with error 16820 (Location16820): 'Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.' on server 127.0.0.1:27017. The full response is { "ok" : 0.0, "errmsg" : "Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.", "code" : 16820, "codeName" : "Location16820" }
       
      How to fix allowDiskUse:true? 
      Thanks

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            tchaouki Chaouki Trabelsi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: