[KAFKA-384] Mongo Source Connector Enhancement Request Created: 31/Jul/23  Updated: 27/Oct/23  Resolved: 04/Aug/23

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Sanchayan Sen Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File image-2023-07-31-17-06-49-353.png    

 Description   

This is on lines of Jira ticket i have submitted for kafka sink connector ( 

KAFKA-380 Mongo Sink Connector Enhancement Request - MongoDB Jira), this is a separate request for a similar enhancement for Kafka source connector

 

Mongo Source Connector Enhancement Request:-

 

                When the target Mongo DB is unavailable, the mongo sink connector with “error.tolerance = all” writes all the data to dead letter topic.

                The purpose of setting “error.tolerance = all” is to ensure continuous processing of data while at the same time messages leading to data exceptions are moved to dead letter topic.

                                Ex:- duplicate / malformed messages.

 

This enhancement request is expected to resolve the current limitation by providing finer granularity of exception handling and to differentiate between retriable (aka transient) and non-retriable (aka non-transient) errors, and process them accordingly.

Examples for the transient errors are mongo DB unavailable, network/connection issue & typical timeout errors.

Examples for the non-transient errors are data violation errors thrown by Mongo DB, like duplicate key/primary key violations etc.,

 

While encountering transient errors, the connector has to provide multiple retry attempts with time interval between each attempt.

While encountering non-transient errors, the connector should continue to put the message into dead letter topic, and start to process the next message.



 Comments   
Comment by Sanchayan Sen [ 04/Aug/23 ]

Hi Ross

 

thank you for the confirmation. We tested it and indeed the requirement fits in the with features available. This Jira can be closed

Comment by Ross Lawley [ 31/Jul/23 ]

Hi sanchayan.sen@natwest.com,

I'm not 100% sure on the requirements here, but I can confirm the only errors reported to the dead letter queue by the Source connector are data conversion issues. These occur when the MongoDB data cannot be converted to SourceRecords. For example the data does not meet the schema requirements.

The Source connector already has a level or resiliency for transient issues (eg network issues). Underneath it uses the a change stream cursor which has an automatic resume process in case of transient errors.

I hope that helps clarify the situation.

Ross

Comment by PM Bot [ 31/Jul/23 ]

Hi sanchayan.sen@natwest.com, thank you for reporting this issue! The team will look into it and get back to you soon.

Generated at Thu Feb 08 09:06:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.