-
Type: Question
-
Resolution: Works as Designed
-
Priority: Major - P3
-
None
-
Affects Version/s: None
-
Component/s: Source
-
None
This is on lines of Jira ticket i have submitted for kafka sink connector (
KAFKA-380 Mongo Sink Connector Enhancement Request - MongoDB Jira), this is a separate request for a similar enhancement for Kafka source connector
Mongo Source Connector Enhancement Request:-
When the target Mongo DB is unavailable, the mongo sink connector with “error.tolerance = all” writes all the data to dead letter topic.
The purpose of setting “error.tolerance = all” is to ensure continuous processing of data while at the same time messages leading to data exceptions are moved to dead letter topic.
Ex:- duplicate / malformed messages.
This enhancement request is expected to resolve the current limitation by providing finer granularity of exception handling and to differentiate between retriable (aka transient) and non-retriable (aka non-transient) errors, and process them accordingly.
Examples for the transient errors are mongo DB unavailable, network/connection issue & typical timeout errors.
Examples for the non-transient errors are data violation errors thrown by Mongo DB, like duplicate key/primary key violations etc.,
While encountering transient errors, the connector has to provide multiple retry attempts with time interval between each attempt.
While encountering non-transient errors, the connector should continue to put the message into dead letter topic, and start to process the next message.