Uploaded image for project: 'Spark Connector'
  1. Spark Connector
  2. SPARK-440

Support `mongodb.keep_alive_ms` system property

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Unknown Unknown
    • 10.4.1
    • Affects Version/s: None
    • Component/s: Performance
    • None
    • Java Drivers
    • Not Needed

      The documentation at https://www.mongodb.com/docs/spark-connector/master/faq/ specifies to set mongodb.keep_alive_ms but in the code the system property is prefixed with spark. This changes makes the code consistent with the documentation while maintaining backwards compatibility.

      When attempting to set this with --conf spark.executor.extraJavaOptions="-Dspark.mongodb.keep_alive_ms=70000" an exception occurs since Spark does not allow setting spark flags with that mechanism on executors:

      java.lang.Exception: spark.executor.extraJavaOptions is not allowed to set Spark
      options (was '-Dspark.mongodb.keep_alive_ms=70000’). Set them directly on a
      SparkConf or in a properties file when using ./bin/spark-submit.
      

      Work-around: Rather than setting the system property via extraJavaOptions on executors, the environment variable JAVA_TOOL_OPTIONS can be used:

      --conf spark.driver.extraJavaOptions=-Dspark.mongodb.keep_alive_ms=70000 
      --conf spark.executorEnv.JAVA_TOOL_OPTIONS=-Dspark.mongodb.keep_alive_ms=70000
      

      PR: https://github.com/mongodb/mongo-spark/pull/128

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            ross@mongodb.com Ross Lawley
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: