Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-81

How to provide allowDiskUse : true option with pipeline of JavaMongoRDD in Java code or any other better approach

    XMLWordPrintable

Details

    • New Feature
    • Status: Closed
    • Major - P3
    • Resolution: Fixed
    • None
    • 2.0.0-rc1, 2.0.0
    • API, Configuration
    • None

    Description

      I am facing the issue "exception: Exceeded memory limit, Pass allowDiskUse:true to opt in." for the example code

      JavaMongoRDD aggregatedRdd = rdd .withPipeline(Arrays.asList(Document.parse(aggregateQuery), Document.parse("{$sort:

      {gatewaytime : -1}

      ")));
      There are many examples to pass the allowDiskUse:true in mongo db aggregate() method or via mongo client in java.

      I did't found any example or help how to pass these type of options using Spark RDD.withPipeline() method or there is any alternate approach better than using allowDiskUse:true.

      Thanks in advance.

      Attachments

        Activity

          People

            Unassigned Unassigned
            sampatkhatri421 Samapt Singh
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: