Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-81

How to provide allowDiskUse : true option with pipeline of JavaMongoRDD in Java code or any other better approach

    • Type: Icon: New Feature New Feature
    • Resolution: Done
    • Priority: Icon: Major - P3 Major - P3
    • 2.0.0-rc1, 2.0.0
    • Affects Version/s: None
    • Component/s: API, Configuration
    • Labels:
      None

      I am facing the issue "exception: Exceeded memory limit, Pass allowDiskUse:true to opt in." for the example code

      JavaMongoRDD aggregatedRdd = rdd .withPipeline(Arrays.asList(Document.parse(aggregateQuery), Document.parse("{$sort:

      {gatewaytime : -1}

      ")));
      There are many examples to pass the allowDiskUse:true in mongo db aggregate() method or via mongo client in java.

      I did't found any example or help how to pass these type of options using Spark RDD.withPipeline() method or there is any alternate approach better than using allowDiskUse:true.

      Thanks in advance.

            Assignee:
            Unassigned Unassigned
            Reporter:
            sampatkhatri421 Samapt Singh
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: