Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-160

Cannot read dataframe from collection

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Critical - P2
    • Resolution: Duplicate
    • 2.2.0, 2.2.1
    • None
    • API
    • Apache Spark 2.1 & 2.2, Mongo server 3.4.9, Mongo-spark scala 2.2.0 & 2.2.1

    Description

      When reading from a collection into a dataframe I get an fatal exception.
      I do the following to load the dataframe:
      val database: String = "db"
      val collection: String = "collection"
      val readConfig = ReadConfig(Map("uri" -> ("mongodb://mongo/" + database + "." + collection)))
      val sessionDF = spark.read.format("com.mongodb.spark.sql").options(readConfig.asOptions).load()
      println(sessionDF.show())

      When I run the same code with 2.1.0 connector the dataframe shows up without an exception.

      The exception when running:
      java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.TypeCoercion$.findTightestCommonType()Lscala/Function2;

      With trace:
      at com.mongodb.spark.sql.MongoInferSchema$.com$mongodb$spark$sql$MongoInferSchema$$compatibleType(MongoInferSchema.scala:135)
      at com.mongodb.spark.sql.MongoInferSchema$$anonfun$getCompatibleArraySchema$1.apply(MongoInferSchema.scala:279)
      at com.mongodb.spark.sql.MongoInferSchema$$anonfun$getCompatibleArraySchema$1.apply(MongoInferSchema.scala:274)
      at scala.collection.IterableLike$class.takeWhile(IterableLike.scala:160)
      at scala.collection.AbstractIterable.takeWhile(Iterable.scala:54)
      at com.mongodb.spark.sql.MongoInferSchema$.getCompatibleArraySchema(MongoInferSchema.scala:274)
      at com.mongodb.spark.sql.MongoInferSchema$.getSchemaFromArray(MongoInferSchema.scala:257)
      at com.mongodb.spark.sql.MongoInferSchema$.com$mongodb$spark$sql$MongoInferSchema$$getDataType(MongoInferSchema.scala:227)
      at com.mongodb.spark.sql.MongoInferSchema$$anonfun$com$mongodb$spark$sql$MongoInferSchema$$getSchemaFromDocument$1.apply(MongoInferSchema.scala:114)
      at com.mongodb.spark.sql.MongoInferSchema$$anonfun$com$mongodb$spark$sql$MongoInferSchema$$getSchemaFromDocument$1.apply(MongoInferSchema.scala:114)
      at scala.collection.Iterator$class.foreach(Iterator.scala:893)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
      at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
      at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
      at com.mongodb.spark.sql.MongoInferSchema$.com$mongodb$spark$sql$MongoInferSchema$$getSchemaFromDocument(MongoInferSchema.scala:114)
      at com.mongodb.spark.sql.MongoInferSchema$$anonfun$2.apply(MongoInferSchema.scala:78)
      at com.mongodb.spark.sql.MongoInferSchema$$anonfun$2.apply(MongoInferSchema.scala:78)
      at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
      at scala.collection.Iterator$class.foreach(Iterator.scala:893)
      at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
      at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
      at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1336)
      at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
      at scala.collection.AbstractIterator.aggregate(Iterator.scala:1336)
      at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
      at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1136)
      at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$25.apply(RDD.scala:1137)
      at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$25.apply(RDD.scala:1137)
      at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
      at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
      at org.apache.spark.scheduler.Task.run(Task.scala:99)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)

      Attachments

        Issue Links

          Activity

            People

              ross@mongodb.com Ross Lawley
              chris-ras Christiaan Ras
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: