Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-253

Allow Kafka Sink Connector to Execute Unordered Bulk Operations

    XMLWordPrintableJSON

Details

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Major - P3 Major - P3
    • 1.7.0
    • None
    • Sink
    • Needed
    • Hide

      If implemented, this new behavior will need to be properly documented.

      Show
      If implemented, this new behavior will need to be properly documented.

    Description

      Hi Team,

      As of now, the Sink Connector only executes operations using "ordered" bulk writes that guarantee the message ordering within each source topic partition.

      There might be circumstances where ordering is not required and executing "unordered" bulk operations might have benefits:

      • Increase performance due to the ability to parallelize "unordered" bulk write operations.
      • Avoiding the whole batch of messages to fail, specially when the operation that fails is the first one of the "ordered" list.

      The default should still be "ordered" bulk writes but I propose to add the ability to modify this behavior by adding a property to switch to "unordered" bulk operations. The implications of this change should be made very clear in our documentation page and it might even be wise to throw a warning about message processing order in the logs.

      Thanks
      Diego

      Attachments

        Activity

          People

            valentin.kovalenko@mongodb.com Valentin Kavalenka
            diego.rodriguez@mongodb.com Diego Rodriguez (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: