-
Type: Bug
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: 2.1.3, 2.2.4, 2.3.0
-
Component/s: Configuration
-
None
-
Environment:Centos
Hello Team !
I'm using zeppelin Version 0.8.0 ,mongodb 4.0, spark 2.2.0, mongospark connector 2.2.4, mongo java driver 3.8.
I'm trying to load collection from mongodb for visualisation tasks using zeppelin
import com.mongodb.spark.MongoSpark
import com.mongodb.spark.config.{ReadConfig, WriteConfig}
import com.mongodb.spark.sql._
import org.apache.spark.sql.functions._
import org.bson.Document
import collection.JavaConverters._
import org.apache.zeppelin.display.angular.paragraphscope._
import AngularElem._
val readConfig = ReadConfig(Map("uri" > "mongodb://127.0.0.1:27017/", "database" -> "test","collection" -> "Collection_f","allowDiskUse">"True"))
val zipDf = spark.sparkSession.read.mongo(readConfig).toDF()
it gives this error
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 1 times, most recent failure: Lost task 0.0 in stage 2.0 (TID 2, localhost, executor driver): com.mongodb.MongoCommandException: Command failed with error 16820 (Location16820): 'Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.' on server 127.0.0.1:27017. The full response is { "ok" : 0.0, "errmsg" : "Sort exceeded memory limit of 104857600 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.", "code" : 16820, "codeName" : "Location16820" }
How to fix allowDiskUse:true?
Thanks