[KAFKA-380] Mongo Sink Connector Enhancement Request Created: 20/Jul/23  Updated: 24/Jul/23  Resolved: 24/Jul/23

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: New Feature Priority: Major - P3
Reporter: Sanchayan Sen Assignee: Unassigned
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2023-07-20-11-54-28-078.png    
Issue Links:
Cloners
is cloned by KAFKA-374 Implement an error handler to address... Backlog
Related
is related to KAFKA-374 Implement an error handler to address... Backlog

 Description   

Mongo Sink Connector Enhancement Request:-

 

                When the target Mongo DB is unavailable, the mongo sink connector with “error.tolerance = all” writes all the data to dead letter topic.

                The purpose of setting “error.tolerance = all” is to ensure continuous processing of data while at the same time messages leading to data exceptions are moved to dead letter topic.

                                Ex:- duplicate / malformed messages.

 

This enhancement request is expected to resolve the current limitation by providing finer granularity of exception handling and to differentiate between retriable (aka transient) and non-retriable (aka non-transient) errors, and process them accordingly.

Examples for the transient errors are mongo DB unavailable, network/connection issue & typical timeout errors.

Examples for the non-transient errors are data violation errors thrown by Mongo DB, like duplicate key/primary key violations etc.,

 

While encountering transient errors, the connector has to provide multiple retry attempts with time interval between each attempt.

While encountering non-transient errors, the connector should continue to put the message into dead letter topic, and start to process the next message.



 Comments   
Comment by Robert Walters [ 24/Jul/23 ]

Hi sanchayan.sen@natwest.com we have a ticket that will address this for 1.12.  I will reach out to you as we get closer to the implementation to confirm that it meets your requirements.

Comment by PM Bot [ 20/Jul/23 ]

Hi sanchayan.sen@natwest.com, thank you for reporting this issue! The team will look into it and get back to you soon.

Generated at Thu Feb 08 09:06:14 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.