Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-253

Allow Kafka Sink Connector to Execute Unordered Bulk Operations

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.7.0
    • Affects Version/s: None
    • Component/s: Sink
    • Labels:
    • Needed
    • Hide

      If implemented, this new behavior will need to be properly documented.

      Show
      If implemented, this new behavior will need to be properly documented.

      Hi Team,

      As of now, the Sink Connector only executes operations using "ordered" bulk writes that guarantee the message ordering within each source topic partition.

      There might be circumstances where ordering is not required and executing "unordered" bulk operations might have benefits:

      • Increase performance due to the ability to parallelize "unordered" bulk write operations.
      • Avoiding the whole batch of messages to fail, specially when the operation that fails is the first one of the "ordered" list.

      The default should still be "ordered" bulk writes but I propose to add the ability to modify this behavior by adding a property to switch to "unordered" bulk operations. The implications of this change should be made very clear in our documentation page and it might even be wise to throw a warning about message processing order in the logs.

      Thanks
      Diego

            Assignee:
            valentin.kovalenko@mongodb.com Valentin Kavalenka
            Reporter:
            diego.rodriguez@mongodb.com Diego Rodriguez (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: