[KAFKA-361] MongoDB Source Connector - configuration validation runs before replacement when using a Config Provider Created: 15/Mar/23  Updated: 06/Jun/23  Resolved: 20/Mar/23

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: 1.9.1
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Lincoln D'Mello Assignee: Ross Lawley
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
causes KAFKA-371 MongoDB Kafka Connect Connectors - co... Closed
Related
related to KAFKA-322 External secrets via ConfigProvider d... Closed

 Description   

Environment:
Confluent Stack running using Docker Compose (Kafka, KafkaConnect, Zookeeper, etc.) Version: 7.3.0
ConfigProvider being used: SecretsManagerConfigProvider - https://github.com/jcustenborder/kafka-config-provider-aws
Mongo Source Connector being used: https://www.confluent.io/hub/mongodb/kafka-connect-mongodb

Problem:
When using the Kafka Connect REST API to either validate or create a new MongoSourceConnector instance when using the above configuration provider to replace the values of secret configuration items in the configuration as below. I get the error - Invalid value "${secretmanager:dev/db/mongo/test/connection:uri}" for configuration connection.uri

Additionally, I am using an AWS Profile to provide the AWS credentials and I can confirm that I can fetch the secrets from AWS Secrets Manager just fine via the AWS CLI for the profile being used. The AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE environment variables have been set on the Kafka Connect container as well. Also, the Mongo Source connector works just fine when not using a config provider and with a directly supplied connection.url value.

 {
  "connector.class" : "com.mongodb.kafka.connect.MongoSourceConnector",
  "config.providers" : "secretmanager",
  "config.providers.secretmanager.class"  : "com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider",
  "config.providers.secretmanager.param.aws.region" : "us-east-1",
  "tasks.max" : 1,
  "key.converter" : "io.confluent.connect.avro.AvroConverter",
  "value.converter" : "io.confluent.connect.avro.AvroConverter",
  "errors.tolerance" : "all",
  "errors.log.enable": "true",
  "errors.log.include.messages" : "false",
  "errors.deadletterqueue.context.headers.enable": "true",
  "errors.deadletterqueue.topic.name" : "dlq",
  "connection.uri" : "${secretmanager:dev/db/mongo/test/connection:uri}"
}

Could this issue be linked to, or a regression of, #737 ?



 Comments   
Comment by Ross Lawley [ 06/Jun/23 ]

Hi lincoln42@hotmail.com,

I'll respond in KAFKA-371 as well to aid future discovery.

Please ensure that the Kafka workers are configured correctly to use a ConfigProvider as its the worker configuration and not the connector configuration that has to declare the provider:

See: kafka-config-provider-aws documentation

Worker Configuration
Add the following configuration to your Connect worker config file to enable this ConfigProvider to be used by connectors.

config.providers.secretManager.param.secret.prefix=staging
config.providers.secretManager.class=com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider
config.providers=secretManager
config.providers.secretManager.param.aws.region=us-west-2

Connector Configuration
Add the following configurations to your connector configuration when you would like to retrieve a value from the config provider

{
  "username" : "${secretManager:secret/test/some/connector:username}",
  "password" : "${secretManager:secret/test/some/connector:password}"
}

I hope that helps,

Ross

Comment by Lincoln D'Mello [ 12/May/23 ]

robert.walters@mongodb.com I have tested the latest version of the connectors with the above config provider and the problem still persists. However, this time around when I validate the connector configuration with Kafka Connect via the REST validation endpoint, I can see that, in the JSON validation error that is returned from Kakfa Connect, for the 'connection.uri'  config property, the 'value.value' JSON property has the correct uri value that the config provider has retrieved from AWS but somehow the retrieved value has still not replaced the placeholder 

"${secretmanager:dev/db/mongo/test/connection:uri}"

in the actual configuration that Kafka Connect is trying to use to configure the connector instance.
   When I try to create the connector using the above config, I just get an error saying that the 'connection.uri' has an invalid value.
   I would request that this ticket be reopened so this issue can be properly addressed.

Comment by Robert Walters [ 20/Mar/23 ]

PR for fix will be in next version of connector due out at the end of this month (March)

Generated at Thu Feb 08 09:06:12 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.