[DOCS-13757] [KAFKA] Add delivery guarantee to source and sink Created: 09/Jul/20  Updated: 29/Oct/23  Resolved: 11/Sep/20

Status: Closed
Project: Documentation
Component/s: Kafka
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Robert Walters Assignee: Christopher Cho
Resolution: Fixed Votes: 0
Labels: kafka-connect
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
is documented by KAFKA-79 Add the fact that the source guarante... Closed
Duplicate
is duplicated by KAFKA-79 Add the fact that the source guarante... Closed
Participants:
Days since reply: 3 years, 21 weeks, 6 days ago
Epic Link: DOCSP-11739
Story Points: 1

 Description   

Description

There has been some questions around delivery guarantee of data to and from the topic with respect to exactly once, at least once, etc. We should add this delivery information explicitly to the documentation for both source and sink.

Answered by Robert in the forums here: https://developer.mongodb.com/community/forums/t/kafka-connector-delivery-guarantees/8757

For sink default is At Least Once. If there is an error when processing data from a topic the connector will retry the write. However, if the data on the topic contains a unique attribute, it is possible to achieve exactly once semantics by configuring the Sink connector to use upserts and the DocumentIdAdder strategy. The sink connector can not support at most once.

For source default is At least once. There is a risk of duplicate messages if you use copy.existing. Note that change stream events are idempotent so the need to support other delivery guarantees are not applicable.

Scope of changes

All versions

Impact to Other Docs

MVP (Work and Date)

Resources (Scope or Design Docs, Invision, etc.)



 Comments   
Comment by Christopher Cho [ 10/Sep/20 ]

PR ready for review: https://github.com/10gen/docs-kafka-connector/pull/44

Comment by Christopher Cho [ 08/Sep/20 ]

robert.walters quick question:

When you mention that you need to use "upserts and the DocumentIdAdder strategy" to get an exactly-once guarantee, what configuration controls whether it's upserts?

Generated at Thu Feb 08 08:08:39 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.