[KAFKA-384] Mongo Source Connector Enhancement Request Created: 31/Jul/23 Updated: 27/Oct/23 Resolved: 04/Aug/23 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | Source |
| Affects Version/s: | None |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Sanchayan Sen | Assignee: | Ross Lawley |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Attachments: |
|
| Description |
|
This is on lines of Jira ticket i have submitted for kafka sink connector (
Mongo Source Connector Enhancement Request:-
When the target Mongo DB is unavailable, the mongo sink connector with “error.tolerance = all” writes all the data to dead letter topic. The purpose of setting “error.tolerance = all” is to ensure continuous processing of data while at the same time messages leading to data exceptions are moved to dead letter topic. Ex:- duplicate / malformed messages.
This enhancement request is expected to resolve the current limitation by providing finer granularity of exception handling and to differentiate between retriable (aka transient) and non-retriable (aka non-transient) errors, and process them accordingly. Examples for the transient errors are mongo DB unavailable, network/connection issue & typical timeout errors. Examples for the non-transient errors are data violation errors thrown by Mongo DB, like duplicate key/primary key violations etc.,
While encountering transient errors, the connector has to provide multiple retry attempts with time interval between each attempt. While encountering non-transient errors, the connector should continue to put the message into dead letter topic, and start to process the next message. |
| Comments |
| Comment by Sanchayan Sen [ 04/Aug/23 ] |
|
Hi Ross
thank you for the confirmation. We tested it and indeed the requirement fits in the with features available. This Jira can be closed |
| Comment by Ross Lawley [ 31/Jul/23 ] |
|
I'm not 100% sure on the requirements here, but I can confirm the only errors reported to the dead letter queue by the Source connector are data conversion issues. These occur when the MongoDB data cannot be converted to SourceRecords. For example the data does not meet the schema requirements. The Source connector already has a level or resiliency for transient issues (eg network issues). Underneath it uses the a change stream cursor which has an automatic resume process in case of transient errors. I hope that helps clarify the situation. Ross |
| Comment by PM Bot [ 31/Jul/23 ] |
|
Hi sanchayan.sen@natwest.com, thank you for reporting this issue! The team will look into it and get back to you soon. |