[KAFKA-215] New names for errors tolerance config properties Created: 01/Apr/21  Updated: 28/Oct/23  Resolved: 10/Jun/21

Status: Closed
Project: Kafka Connector
Component/s: Sink, Source
Affects Version/s: 1.5.0
Fix Version/s: 1.6.0

Type: Improvement Priority: Critical - P2
Reporter: Chris Egerton Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
Related
related to KAFKA-177 MongoDB Sink Connector deadletterqueu... Closed
related to KAFKA-105 Support errors.tolerance Closed
Documentation Changes: Needed
Documentation Changes Summary:

Document any new properties introduced as a result of this issue, and expand documentation to include behavioral changes to any existing properties.


 Description   

The errors.tolerance, errors.log.enable, and errors.deadletterqueue.topic.name properties introduced as part of the work for KAFKA-78 and KAFKA-105 conflict with the names of properties used by the Connect framework. This prevents users from configuring connectors with different values for the framework-level errors.tolerance and the connector-level errors.tolerance properties, for example, and can be problematic in environments where error tolerance is only enabled with the expectation that faulty records will be sent to a DLQ topic. Since no DLQ topic is supported by the sink connector itself (and likely won't be until the ErrorReporter mechanism introduced in KIP-610 is leveraged), this forces users to choose between potential data loss (by setting errors.tolerance to all) and having the connector die on skip records that fail during the conversion/transformation phase (by setting errors.tolerance to none).

It'd be great if it were possible to set the framework-level errors.tolerance property and related properties to different values than the ones recognized by the connector. Implementation-wise, a couple of options include:

  1. Rename the existing configuration properties. For example, errors.tolerance might be changed to mongo.errors.tolerance.
  2. Add new properties with different names that take precedence over the existing properties, if specified. Going with the above example, if someone specified only errors.tolerance = ALL, then all errors would be tolerated; however, if someone specified both errors.tolerance = ALL and mongo.errors.tolerance = NONE, then no errors would be tolerated as the latter would take priority over the former.
  3. Like option 2, add new properties with different names that take precedence over the existing properties if specified, but also deprecate the existing properties and note that they will be removed in an upcoming release.

Filed as critical (P2) priority as this has the potential right now to lead to unexpected data loss for users of the connector.



 Comments   
Comment by Ross Lawley [ 10/Jun/21 ]

Thanks chrise@confluent.io this has been merged into master and is scheduled for the 1.6.0 release.

Comment by Githook User [ 10/Jun/21 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Updated changelog

Added mongo specific override options for error handling properties

KAFKA-215
Branch: master
https://github.com/mongodb/mongo-kafka/commit/ececd75d021d3c537ee474a00daa12f908e88e8f

Comment by Githook User [ 10/Jun/21 ]

Author:

{'name': 'Chris Egerton', 'email': 'chrise@confluent.io', 'username': 'C0urante'}

Message: KAFKA-215: Add override options for error handling properties
Branch: master
https://github.com/mongodb/mongo-kafka/commit/3969d115d3e96d554aa43b009c58253782cd5ab8

Comment by Chris Egerton [ 01/Apr/21 ]

Ross Lawley--thoughts? Happy to do the PR legwork as long as this has some traction and is recognized as a worthwhile change.

Generated at Thu Feb 08 09:05:51 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.