[KAFKA-40] Support schema for Source connector Created: 09/Jul/19 Updated: 22/Sep/21 Resolved: 08/Sep/20 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | 1.3.0 |
| Type: | Epic | Priority: | Major - P3 |
| Reporter: | Scott L'Hommedieu (Inactive) | Assignee: | Ross Lawley |
| Resolution: | Done | Votes: | 12 |
| Labels: | FY21Q2 | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||||||
| Case: | (copied to CRM) | ||||||||||||||||||||
| Start date: | |||||||||||||||||||||
| End date: | |||||||||||||||||||||
| Calendar Time: | 7 weeks, 4 days | ||||||||||||||||||||
| Detailed Project Statuses: | Engineer(s): Ross 2020-08-25:
2020-07-28: Initial target of 2020-08-21 (6 Weeks) Goals for the next month:
Completed work:
|
||||||||||||||||||||
| Description |
|
Epic Summary SummaryCurrently the source connector converts all change stream documents to extended Json strings before publishing to a topic. Change stream events have a defined schema as such but contain a number of optional fields:
The main complexities will be deciding how to handle dynamic document data: fullDocument, documentKey and updateDescription.updatedFields.
A secondary aim will be to ensure users can access data from the ChangeStream in a way that is easily consumable outside of MongoDB: |
| Comments |
| Comment by Jeffrey Cheak [ 25/Aug/20 ] | ||||||||||||||||
|
Can you clarify which Jeff? | ||||||||||||||||
| Comment by Hamid Jawaid [ 13/Jul/20 ] | ||||||||||||||||
|
Hi Ross, Does this mean, till 1.2 MongoDB Source Connector can not have a key other than the default one(ChangeStreamEvent-ID) as ValueToKey won't work due to error mentioned in this Epic? Default Key: {"_id":{"_data":"<unique-changestream-event-id->"}} Or, we can still have a custom key but need to write our own custom SMT/Converter? eg. I have a nested Document and one of the field-value(unique) I want to use as partition key so that ordering of updates are maintained. My key would be like: fullDocument->userId Is it possible to use custom field(eg fullDocument->userId) as partition key using custom SMT/Converter in MongoDB Source Connector? | ||||||||||||||||
| Comment by Ludovic Dussart [ 19/May/20 ] | ||||||||||||||||
|
Related to this limitation, it is not possible to use Kafka Simple Message Transformation because SMT need to manipulate a STRUCT (and not a String). Even if io.confluent.connect.avro.AvroConverter is used for key or value, the type in declared as String, resulting by a Kafka exception :
Related sink connector configuration :
|