-
Type:
Improvement
-
Resolution: Fixed
-
Priority:
Unknown
-
Affects Version/s: None
-
Component/s: Performance
-
None
-
Java Drivers
-
Not Needed
The documentation at https://www.mongodb.com/docs/spark-connector/master/faq/ specifies to set mongodb.keep_alive_ms but in the code the system property is prefixed with spark. This changes makes the code consistent with the documentation while maintaining backwards compatibility.
When attempting to set this with --conf spark.executor.extraJavaOptions="-Dspark.mongodb.keep_alive_ms=70000" an exception occurs since Spark does not allow setting spark flags with that mechanism on executors:
java.lang.Exception: spark.executor.extraJavaOptions is not allowed to set Spark options (was '-Dspark.mongodb.keep_alive_ms=70000’). Set them directly on a SparkConf or in a properties file when using ./bin/spark-submit.
Work-around: Rather than setting the system property via extraJavaOptions on executors, the environment variable JAVA_TOOL_OPTIONS can be used:
--conf spark.driver.extraJavaOptions=-Dspark.mongodb.keep_alive_ms=70000 --conf spark.executorEnv.JAVA_TOOL_OPTIONS=-Dspark.mongodb.keep_alive_ms=70000