Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-166

MongoDB sink connector returns `topic` in validation result when not in ConfigDef or in connector properties

    • Type: Icon: Bug Bug
    • Resolution: Fixed
    • Priority: Icon: Major - P3 Major - P3
    • 1.4.0
    • Affects Version/s: 1.2.0, 1.3.0
    • Component/s: Sink
    • None

      When a new connector configuration for the MongoDB sink connector is validated using Connect, the MongoDB sink connector includes in the validation output the `topic` property (with a value matching the `topics` property) even though no such property is defined in the ConfigDef and is not even included in the connector configuration.

      For example, validate a connector configuration as follows:

      curl -X PUT -H "Content-Type: application/json" --data @mongo-sink.json http://localhost:8083/connector-plugins/MongoSinkConnector/config/validate 

      where the mongo-sink.json file contains:

        "name": "my-mongo-sink",
        "document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.KafkaMetaDataStrategy",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter"

      Note that this configuration specified a `topics` (plural) property, but did not specify a `topic` (singular) property. Connect requires either sink connector configurations to specify the topics to be consumed by the connector via a `topics` or `topics.regex` property, but not both.

      The result of the Connect REST API validation is the following JSON:

        "name": "com.mongodb.kafka.connect.MongoSinkConnector",
        "error_count": 0,
        "groups": [
          "Error Handling",
          "Post Processing",
          "Id Strategies",
          "Change Data Capture"
        "configs": [
            "definition": null,
            "value": {
              "name": "topic",
              "value": "connect-offsets",
              "recommended_values": [],
              "errors": [
                "Configuration is not defined: topic"
              "visible": true
            "definition": {
              "name": "topics",
              "type": "LIST",
              "required": false,
              "default_value": "",
              "importance": "HIGH",
              "documentation": "A list of kafka topics for the sink connector, separated by commas",
              "group": "Connection",
              "width": "MEDIUM",
              "display_name": "The Kafka topics",
              "dependents": [],
              "order": 1
            "value": {
              "name": "topics",
              "value": "connect-offsets",
              "recommended_values": [],
              "errors": [],
              "visible": true

      AK Connect is currently marking this as an error because it is not defined in the connector's ConfigDef, but that's a bug (https://issues.apache.org/jira/browse/KAFKA-10600).

      Why does the MongoDB sink connector include in its validation output a `topic` configuration property when it is not required and not even supplied by the user? It's even stranger that the value of this `topic` property matches the value of the user-supplied `topics` property.

      Perhaps the connector at one time looked for the `topic` property, and keeping that helps maintain backward compatibility. But if this is the only reason, then perhapsĀ this should be transparent to users and the `MongoSinkConnector.validate(...)` method should not include `topic` in its response.

            ross@mongodb.com Ross Lawley
            rhauch@gmail.com Randall Hauch
            1 Vote for this issue
            3 Start watching this issue