[KAFKA-361] MongoDB Source Connector - configuration validation runs before replacement when using a Config Provider Created: 15/Mar/23 Updated: 06/Jun/23 Resolved: 20/Mar/23 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | Source |
| Affects Version/s: | 1.9.1 |
| Fix Version/s: | None |
| Type: | Question | Priority: | Major - P3 |
| Reporter: | Lincoln D'Mello | Assignee: | Ross Lawley |
| Resolution: | Duplicate | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||||||||||
| Description |
|
Environment: Problem: Additionally, I am using an AWS Profile to provide the AWS credentials and I can confirm that I can fetch the secrets from AWS Secrets Manager just fine via the AWS CLI for the profile being used. The AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE environment variables have been set on the Kafka Connect container as well. Also, the Mongo Source connector works just fine when not using a config provider and with a directly supplied connection.url value.
Could this issue be linked to, or a regression of, #737 ? |
| Comments |
| Comment by Ross Lawley [ 06/Jun/23 ] | ||||||||
|
I'll respond in Please ensure that the Kafka workers are configured correctly to use a ConfigProvider as its the worker configuration and not the connector configuration that has to declare the provider: See: kafka-config-provider-aws documentation
I hope that helps, Ross | ||||||||
| Comment by Lincoln D'Mello [ 12/May/23 ] | ||||||||
|
robert.walters@mongodb.com I have tested the latest version of the connectors with the above config provider and the problem still persists. However, this time around when I validate the connector configuration with Kafka Connect via the REST validation endpoint, I can see that, in the JSON validation error that is returned from Kakfa Connect, for the 'connection.uri' config property, the 'value.value' JSON property has the correct uri value that the config provider has retrieved from AWS but somehow the retrieved value has still not replaced the placeholder
in the actual configuration that Kafka Connect is trying to use to configure the connector instance. | ||||||||
| Comment by Robert Walters [ 20/Mar/23 ] | ||||||||
|
PR for fix will be in next version of connector due out at the end of this month (March) |