[KAFKA-166] MongoDB sink connector returns `topic` in validation result when not in ConfigDef or in connector properties Created: 12/Oct/20  Updated: 28/Oct/23  Resolved: 13/Jan/21

Status: Closed
Project: Kafka Connector
Component/s: Sink
Affects Version/s: 1.2.0, 1.3.0
Fix Version/s: 1.4.0

Type: Bug Priority: Major - P3
Reporter: Randall Hauch Assignee: Ross Lawley
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to KAFKA-195 Sink don't validate topic config when... Closed

 Description   

When a new connector configuration for the MongoDB sink connector is validated using Connect, the MongoDB sink connector includes in the validation output the `topic` property (with a value matching the `topics` property) even though no such property is defined in the ConfigDef and is not even included in the connector configuration.

For example, validate a connector configuration as follows:

curl -X PUT -H "Content-Type: application/json" --data @mongo-sink.json http://localhost:8083/connector-plugins/MongoSinkConnector/config/validate 

where the mongo-sink.json file contains:

{
  "name": "my-mongo-sink",
  "connector.class":"com.mongodb.kafka.connect.MongoSinkConnector",
  "tasks.max":1,
    "connection.uri":"mongodb+srv://<user>:<password>@cluster0.abcde.mongodb.net/<dbname>?retryWrites=true&w=majority",
  "database":"sink-test",
  "collection":"docs",
  "topics":"connect-offsets",
  "batch.size":0,
  "document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.KafkaMetaDataStrategy",
  "value.converter":"org.apache.kafka.connect.json.JsonConverter",
  "key.converter": "org.apache.kafka.connect.storage.StringConverter"
}

Note that this configuration specified a `topics` (plural) property, but did not specify a `topic` (singular) property. Connect requires either sink connector configurations to specify the topics to be consumed by the connector via a `topics` or `topics.regex` property, but not both.

The result of the Connect REST API validation is the following JSON:

{
  "name": "com.mongodb.kafka.connect.MongoSinkConnector",
  "error_count": 0,
  "groups": [
    "Common",
    "Transforms",
    "Predicates",
    "Error Handling",
    "Connection",
    "Overrides",
    "Namespace",
    "Writes",
    "Post Processing",
    "Id Strategies",
    "Errors",
    "Change Data Capture"
  ],
  "configs": [
    {
      "definition": null,
      "value": {
        "name": "topic",
        "value": "connect-offsets",
        "recommended_values": [],
        "errors": [
          "Configuration is not defined: topic"
        ],
        "visible": true
      }
    },
    ...
    {
      "definition": {
        "name": "topics",
        "type": "LIST",
        "required": false,
        "default_value": "",
        "importance": "HIGH",
        "documentation": "A list of kafka topics for the sink connector, separated by commas",
        "group": "Connection",
        "width": "MEDIUM",
        "display_name": "The Kafka topics",
        "dependents": [],
        "order": 1
      },
      "value": {
        "name": "topics",
        "value": "connect-offsets",
        "recommended_values": [],
        "errors": [],
        "visible": true
      }
    },
    ...
  ]
} 

AK Connect is currently marking this as an error because it is not defined in the connector's ConfigDef, but that's a bug (https://issues.apache.org/jira/browse/KAFKA-10600).

Why does the MongoDB sink connector include in its validation output a `topic` configuration property when it is not required and not even supplied by the user? It's even stranger that the value of this `topic` property matches the value of the user-supplied `topics` property.

Perhaps the connector at one time looked for the `topic` property, and keeping that helps maintain backward compatibility. But if this is the only reason, then perhapsĀ this should be transparent to users and the `MongoSinkConnector.validate(...)` method should not include `topic` in its response.



 Comments   
Comment by Githook User [ 13/Jan/21 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Fixed sink validation issue including synthetic config property (#47)

KAFKA-166
Branch: master
https://github.com/mongodb/mongo-kafka/commit/3085c9cb1ff4f943341fa29da43d1cc2976e5f5c

Comment by Ross Lawley [ 12/Jan/21 ]

https://github.com/mongodb/mongo-kafka/pull/47

Comment by Randall Hauch [ 12/Oct/20 ]

Consider downgrading the priority for this; it's more of a suggestion to minimize effects on Connect-based tooling.

Generated at Thu Feb 08 09:05:44 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.