[KAFKA-269] Write mode to Embed Entities Created: 08/Dec/21 Updated: 17/Jul/23 |
|
| Status: | Backlog |
| Project: | Kafka Connector |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Juan Soto (Inactive) | Assignee: | Unassigned |
| Resolution: | Unresolved | Votes: | 1 |
| Labels: | internal-user | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Description |
|
Hello team, At the moment kafka connect only works with relational models or where you can merge multiple streams in one document.
If we want to work with model like the picture attached. There is no write model that supports embed entities.
Is an array the best approach for kafka connect? When you design a kafka stream you need to create idempotent flows (as much as possible). If kafka rebalance is it possible that your connector pushes two times the same item. For this reason, the best schema in my opinion is
The upsert operation that manages this schema is the following.
At the moment the new write mode is not supporting new connector properties. But this will be some of them:
This new write mode is using some of the functionality provided by UpdateOneBusinessKeyTimestampStrategy Regards, Juan
|