Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-81

How to provide allowDiskUse : true option with pipeline of JavaMongoRDD in Java code or any other better approach

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0-rc1, 2.0.0
    • Component/s: API, Configuration
    • Labels:
      None
    • # Replies:
      8
    • Last comment by Customer:
      false
    • Sprint:
      JVM Sprint 36

      Description

      I am facing the issue "exception: Exceeded memory limit, Pass allowDiskUse:true to opt in." for the example code

      JavaMongoRDD aggregatedRdd = rdd .withPipeline(Arrays.asList(Document.parse(aggregateQuery), Document.parse("{$sort:

      {gatewaytime : -1}

      ")));
      There are many examples to pass the allowDiskUse:true in mongo db aggregate() method or via mongo client in java.

      I did't found any example or help how to pass these type of options using Spark RDD.withPipeline() method or there is any alternate approach better than using allowDiskUse:true.

      Thanks in advance.

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since reply:
                3 years, 12 weeks, 5 days ago
                Date of 1st Reply: