Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-215

New names for errors tolerance config properties

    • Type: Icon: Improvement Improvement
    • Resolution: Fixed
    • Priority: Icon: Critical - P2 Critical - P2
    • 1.6.0
    • Affects Version/s: 1.5.0
    • Component/s: Sink, Source
    • None
    • Needed
    • Hide

      Document any new properties introduced as a result of this issue, and expand documentation to include behavioral changes to any existing properties.

      Show
      Document any new properties introduced as a result of this issue, and expand documentation to include behavioral changes to any existing properties.

      The errors.tolerance, errors.log.enable, and errors.deadletterqueue.topic.name properties introduced as part of the work for KAFKA-78 and KAFKA-105 conflict with the names of properties used by the Connect framework. This prevents users from configuring connectors with different values for the framework-level errors.tolerance and the connector-level errors.tolerance properties, for example, and can be problematic in environments where error tolerance is only enabled with the expectation that faulty records will be sent to a DLQ topic. Since no DLQ topic is supported by the sink connector itself (and likely won't be until the ErrorReporter mechanism introduced in KIP-610 is leveraged), this forces users to choose between potential data loss (by setting errors.tolerance to all) and having the connector die on skip records that fail during the conversion/transformation phase (by setting errors.tolerance to none).

      It'd be great if it were possible to set the framework-level errors.tolerance property and related properties to different values than the ones recognized by the connector. Implementation-wise, a couple of options include:

      1. Rename the existing configuration properties. For example, errors.tolerance might be changed to mongo.errors.tolerance.
      2. Add new properties with different names that take precedence over the existing properties, if specified. Going with the above example, if someone specified only errors.tolerance = ALL, then all errors would be tolerated; however, if someone specified both errors.tolerance = ALL and mongo.errors.tolerance = NONE, then no errors would be tolerated as the latter would take priority over the former.
      3. Like option 2, add new properties with different names that take precedence over the existing properties if specified, but also deprecate the existing properties and note that they will be removed in an upcoming release.

      Filed as critical (P2) priority as this has the potential right now to lead to unexpected data loss for users of the connector.

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            chrise@confluent.io Chris Egerton
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: