Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-250

with mongodb connector, spark stuck at the last task.

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Done
    • Affects Version/s: 2.3.2
    • Fix Version/s: None
    • Component/s: Reads
    • Labels:
      None
    • Environment:
      linux, mongodb3.x, spark2.3.1, scala2.11.11

      Description

      //
      val rdd = Global.sparkContext.loadFromMongoDB(...).withPipeline(...)
       
      rdd.count()

      it always stuck at the last task. it may take 30 minutes to finish this last task, or maybe hange foreaver. No exception or error is found.

       

      If it just reads few records, for example, 2000 records, it could finish the last task quickly. If it reads above 100000 records, it will hange there.

      I have set 

      System.setProperty("spark.mongodb.keep_alive_ms", "1000000"), but it does not help.

       

        Attachments

          Activity

            People

            Assignee:
            ross.lawley Ross Lawley
            Reporter:
            efenzha Feng Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: