Uploaded image for project: 'Documentation'
  1. Documentation
  2. DOCS-13757

[KAFKA] Add delivery guarantee to source and sink

    XMLWordPrintable

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Kafka
    • Labels:

      Description

      Description

      There has been some questions around delivery guarantee of data to and from the topic with respect to exactly once, at least once, etc. We should add this delivery information explicitly to the documentation for both source and sink.

      Answered by Robert in the forums here: https://developer.mongodb.com/community/forums/t/kafka-connector-delivery-guarantees/8757

      For sink default is At Least Once. If there is an error when processing data from a topic the connector will retry the write. However, if the data on the topic contains a unique attribute, it is possible to achieve exactly once semantics by configuring the Sink connector to use upserts and the DocumentIdAdder strategy. The sink connector can not support at most once.

      For source default is At least once. There is a risk of duplicate messages if you use copy.existing. Note that change stream events are idempotent so the need to support other delivery guarantees are not applicable.

      Scope of changes

      All versions

      Impact to Other Docs

      MVP (Work and Date)

      Resources (Scope or Design Docs, Invision, etc.)

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              chris.cho Christopher Cho
              Reporter:
              robert.walters Robert Walters
              Participants:
              Last commenter:
              Ross Lawley Ross Lawley
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Days since reply:
                1 year, 6 weeks, 4 days ago
                Date of 1st Reply: