Allow Kafka Sink Connector to Execute Unordered Bulk Operations

XMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Major - P3
    • 1.7.0
    • Affects Version/s: None
    • Component/s: Sink
    • Needed
    • Hide

      If implemented, this new behavior will need to be properly documented.

      Show
      If implemented, this new behavior will need to be properly documented.
    • None
    • None
    • None
    • None
    • None
    • None

      Hi Team,

      As of now, the Sink Connector only executes operations using "ordered" bulk writes that guarantee the message ordering within each source topic partition.

      There might be circumstances where ordering is not required and executing "unordered" bulk operations might have benefits:

      • Increase performance due to the ability to parallelize "unordered" bulk write operations.
      • Avoiding the whole batch of messages to fail, specially when the operation that fails is the first one of the "ordered" list.

      The default should still be "ordered" bulk writes but I propose to add the ability to modify this behavior by adding a property to switch to "unordered" bulk operations. The implications of this change should be made very clear in our documentation page and it might even be wise to throw a warning about message processing order in the logs.

      Thanks
      Diego

            Assignee:
            Valentin Kavalenka
            Reporter:
            Diego Rodriguez (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: