Description
When a new connector configuration for the MongoDB sink connector is validated using Connect, the MongoDB sink connector includes in the validation output the `topic` property (with a value matching the `topics` property) even though no such property is defined in the ConfigDef and is not even included in the connector configuration.
For example, validate a connector configuration as follows:
curl -X PUT -H "Content-Type: application/json" --data @mongo-sink.json http://localhost:8083/connector-plugins/MongoSinkConnector/config/validate
|
where the mongo-sink.json file contains:
{
|
"name": "my-mongo-sink", |
"connector.class":"com.mongodb.kafka.connect.MongoSinkConnector", |
"tasks.max":1, |
"connection.uri":"mongodb+srv://<user>:<password>@cluster0.abcde.mongodb.net/<dbname>?retryWrites=true&w=majority", |
"database":"sink-test", |
"collection":"docs", |
"topics":"connect-offsets", |
"batch.size":0, |
"document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.KafkaMetaDataStrategy", |
"value.converter":"org.apache.kafka.connect.json.JsonConverter", |
"key.converter": "org.apache.kafka.connect.storage.StringConverter" |
}
|
Note that this configuration specified a `topics` (plural) property, but did not specify a `topic` (singular) property. Connect requires either sink connector configurations to specify the topics to be consumed by the connector via a `topics` or `topics.regex` property, but not both.
The result of the Connect REST API validation is the following JSON:
{
|
"name": "com.mongodb.kafka.connect.MongoSinkConnector", |
"error_count": 0, |
"groups": [ |
"Common", |
"Transforms", |
"Predicates", |
"Error Handling", |
"Connection", |
"Overrides", |
"Namespace", |
"Writes", |
"Post Processing", |
"Id Strategies", |
"Errors", |
"Change Data Capture" |
],
|
"configs": [ |
{
|
"definition": null, |
"value": { |
"name": "topic", |
"value": "connect-offsets", |
"recommended_values": [], |
"errors": [ |
"Configuration is not defined: topic" |
],
|
"visible": true |
}
|
},
|
...
|
{
|
"definition": { |
"name": "topics", |
"type": "LIST", |
"required": false, |
"default_value": "", |
"importance": "HIGH", |
"documentation": "A list of kafka topics for the sink connector, separated by commas", |
"group": "Connection", |
"width": "MEDIUM", |
"display_name": "The Kafka topics", |
"dependents": [], |
"order": 1 |
},
|
"value": { |
"name": "topics", |
"value": "connect-offsets", |
"recommended_values": [], |
"errors": [], |
"visible": true |
}
|
},
|
...
|
]
|
}
|
AK Connect is currently marking this as an error because it is not defined in the connector's ConfigDef, but that's a bug (https://issues.apache.org/jira/browse/KAFKA-10600).
Why does the MongoDB sink connector include in its validation output a `topic` configuration property when it is not required and not even supplied by the user? It's even stranger that the value of this `topic` property matches the value of the user-supplied `topics` property.
Perhaps the connector at one time looked for the `topic` property, and keeping that helps maintain backward compatibility. But if this is the only reason, then perhaps this should be transparent to users and the `MongoSinkConnector.validate(...)` method should not include `topic` in its response.
Attachments
Issue Links
- related to
-
KAFKA-195 Sink don't validate topic config when using topics.regex
-
- Closed
-