[DOCS-12790] Spark Connector - new releases Created: 11/Jun/19  Updated: 29/Oct/23  Resolved: 05/Aug/19

Status: Closed
Project: Documentation
Component/s: Spark Connector
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Ross Lawley Assignee: Jonathan DeStefano
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

https://docs.mongodb.com/spark-connector/current/


Participants:
Days since reply: 4 years, 27 weeks, 2 days ago

 Description   

Description

New releases

2.4.1, 2.3.3, 2.2.7, 2.1.6

ChangeLog (same for all releases)

  • Ensure nullable fields or container types accept null values
  • Added ReadConfig.batchSize property
  • Renamed system property spark.mongodb.keep_alive_ms to mongodb.keep_alive_ms
  • Added MongoDriverInformation to the default MongoClient
  • Update to latest Java driver (3.10.+)
  • Update PartitionerHelper.matchQuery - no longer includes $ne/$exists checks
  • Added logging of partitioner and their queries
  • Added WriteConfig.extendedBsonTypes setting, so users can disable extended bson types when writing.
  • Added Java spi can now use short form: spark.read.format("mongo")

Source: https://github.com/mongodb/mongo-spark/blob/master/doc/7-Changelog.md#241

New Configuration Options:

Input Configuration:

  • batchSize - the optional size for the internal batches used within the cursor

    Output Configuration

  • extendedBsonTypes - enables extended Bson Types when writing data to MongoDB. Default: true

    Cache Configuration

  • spark.mongodb.keep_alive_ms renamed to mongodb.keep_alive_ms

Other changes

Can now use: spark.read.format("mongo")

  • Replace all instances of: format("com.mongodb.spark.sql") with format("mongo")
  • Replace all instances of: format("com.mongodb.spark.sql.DefaultSource") with format("mongo")

Scope of changes

Impact to Other Docs

MVP (Work and Date)

Resources (Scope or Design Docs, Invision, etc.)



 Comments   
Comment by Jonathan DeStefano [ 05/Aug/19 ]

Merged and published.

Comment by Jonathan DeStefano [ 18/Jul/19 ]

ross.lawley The following PR is ready for review at your convenience:

https://github.com/mongodb/docs-spark-connector/pull/32

Once approved, I can backport the changes to the other doc versions.

Generated at Thu Feb 08 08:06:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.