-
Type: Improvement
-
Resolution: Fixed
-
Priority: Major - P3
-
Affects Version/s: None
-
Component/s: Sink
-
(copied to CRM)
-
Needed
-
Hi Team,
As of now, the Sink Connector only executes operations using "ordered" bulk writes that guarantee the message ordering within each source topic partition.
There might be circumstances where ordering is not required and executing "unordered" bulk operations might have benefits:
- Increase performance due to the ability to parallelize "unordered" bulk write operations.
- Avoiding the whole batch of messages to fail, specially when the operation that fails is the first one of the "ordered" list.
The default should still be "ordered" bulk writes but I propose to add the ability to modify this behavior by adding a property to switch to "unordered" bulk operations. The implications of this change should be made very clear in our documentation page and it might even be wise to throw a warning about message processing order in the logs.
Thanks
Diego
- links to