Summary

Migrating from original Spark Connector to V10 is not possible if customers are using RDDs

Motivation

Who is the affected end user?

Spark users who are using Java/Scala and use RDDs

How does this affect the end user?

They are unblocked from upgrading to the latest version of the Spark Connector

How likely is it that this problem or use case will occur?

If the customer wants to use the new connector they would have to rewrite their spark app to use Datasets 

If the problem does occur, what are the consequences and how severe are they?

N/A

Is this issue urgent?

Not urgent as the Spark V10 has just shipped but as adoption continues this will become more of a blocker

Is this ticket required by a downstream team?

no

Is this ticket only for tests?

N/A

Cast of Characters

Engineering Lead: Jeff Yemin
Document Author: Robert Walters / Ross Lawley
POCers:
Product Owner: Robert Walters
Program Manager:
Stakeholders:

Channels & Docs

Slack Channel

[Scope Document|some.url]

[Technical Design Document|some.url]