DataFrame creation needs explicit schema support

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Done
    • Priority: Major - P3
    • 0.3
    • Affects Version/s: None
    • Component/s: None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Due to restrictions in Scala 2.10 that limit Case Classes to 22 parameters, I tried to create an explicit schema definition and pass it to the MongoRDD's toDF() method and received this error:

      scala> val newTestType = StructType(
     |   StructField("sentClaims", DoubleType, true)::
     |   StructField("totalOrders", DoubleType, true)::
     |   Nil
     | )
newTestType: org.apache.spark.sql.types.StructType = StructType(StructField(sentClaims,DoubleType,true), StructField(totalOrders,DoubleType,true))
scala> val simpleProgrammaticMongoDF = sqlContext.loadFromMongoDB().toDF[newTestType]()
<console>:58: error: not found: type newTestType
         val simpleProgrammaticMongoDF = sqlContext.loadFromMongoDB().toDF[newTestType]()

      Please provide a toDF(schema: StructType) method that will allow me to build and use schemas as needed.

              Assignee:
              Unassigned
              Reporter:
              Richard Williams
              None
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: