[KAFKA-380] Mongo Sink Connector Enhancement Request Created: 20/Jul/23 Updated: 24/Jul/23 Resolved: 24/Jul/23 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | None |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | New Feature | Priority: | Major - P3 |
| Reporter: | Sanchayan Sen | Assignee: | Unassigned |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
||||||||||||||||
| Issue Links: |
|
||||||||||||||||
| Description |
|
Mongo Sink Connector Enhancement Request:-
When the target Mongo DB is unavailable, the mongo sink connector with “error.tolerance = all” writes all the data to dead letter topic. The purpose of setting “error.tolerance = all” is to ensure continuous processing of data while at the same time messages leading to data exceptions are moved to dead letter topic. Ex:- duplicate / malformed messages.
This enhancement request is expected to resolve the current limitation by providing finer granularity of exception handling and to differentiate between retriable (aka transient) and non-retriable (aka non-transient) errors, and process them accordingly. Examples for the transient errors are mongo DB unavailable, network/connection issue & typical timeout errors. Examples for the non-transient errors are data violation errors thrown by Mongo DB, like duplicate key/primary key violations etc.,
While encountering transient errors, the connector has to provide multiple retry attempts with time interval between each attempt. While encountering non-transient errors, the connector should continue to put the message into dead letter topic, and start to process the next message. |
| Comments |
| Comment by Robert Walters [ 24/Jul/23 ] |
|
Hi sanchayan.sen@natwest.com we have a ticket that will address this for 1.12. I will reach out to you as we get closer to the implementation to confirm that it meets your requirements. |
| Comment by PM Bot [ 20/Jul/23 ] |
|
Hi sanchayan.sen@natwest.com, thank you for reporting this issue! The team will look into it and get back to you soon. |