Details
-
Question
-
Resolution: Fixed
-
Critical - P2
-
None
-
None
-
(copied to CRM)
Description
After packaging together with the jar a ConfigProvider (the standard way external secrets are added to Kafka Connect plugins) for the purposes of getting external secrets and configuring the config provider itself (in this case a VaultConfigProvider), e.g.
"config.providers": "vault",
|
"config.providers.vault.class": "com.github.jcustenborder.kafka.config.vault.VaultConfigProvider",
|
"config.providers.vault.param.vault.address": "https://vault.domain.com",
|
"config.providers.vault.param.vault.token": "atokenhere",
|
"config.providers.vault.param.vault.login.by": "Token"
|
// vault host, token and path redacted
using / invoking the use of the secret in the plugin
"connection.uri": "${vault:kv/my/path:MONGO_URI}",
|
will result in the secret engine being created and invoked:
INFO VaultConfigProviderConfig values:
|
vault.address = https://vault.domain.com
|
vault.login.by = Token
|
vault.max.retries = 5
|
vault.namespace =
|
vault.prefix =
|
vault.retry.interval.ms = 2000
|
vault.secret.minimum.ttl.ms = 1000
|
vault.ssl.verify.enabled = true
|
vault.token = [hidden]
|
(com.github.jcustenborder.kafka.config.vault.VaultConfigProviderConfig:376)
|
com.bettercloud.vault.Vault <init>
|
INFO: Constructing a Vault instance with no provided Engine version, defaulting to version 2.
|
INFO Authenticated to Vault as approle: path: auth/approle/login (com.github.jcustenborder.kafka.config.vault.AuthHandlers:131)
|
INFO get() - path = 'my/path' keys = '[MONGO_URI]' (com.github.jcustenborder.kafka.config.vault.VaultConfigProvider:53)
|
INFO VaultConfigProviderConfig values:
|
vault.address = https://vault.domain.com
|
vault.login.by = Token
|
vault.max.retries = 5
|
vault.namespace =
|
vault.prefix =
|
vault.retry.interval.ms = 2000
|
vault.secret.minimum.ttl.ms = 1000
|
vault.ssl.verify.enabled = true
|
vault.token = [hidden]
|
(com.github.jcustenborder.kafka.config.vault.VaultConfigProviderConfig:376)
|
Jul 22, 2022 6:32:41 AM com.bettercloud.vault.Vault <init>
|
INFO: Constructing a Vault instance with no provided Engine version, defaulting to version 2.
|
[2022-07-22 06:32:41,789] INFO Authenticated to Vault as approle: path: auth/approle/login (com.github.jcustenborder.kafka.config.vault.AuthHandlers:131)
|
[2022-07-22 06:32:41,789] INFO get() - path = 'my/path' keys = '[MONGO_URI]' (com.github.jcustenborder.kafka.config.vault.VaultConfigProvider:53)
|
[2022-07-22 06:32:42,009] INFO AbstractConfig values:
|
(org.apache.kafka.common.config.AbstractConfig:376)
|
but then Kafka Connect, and maybe more specifically the mongo plugin throws this error:
{"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nInvalid value ${vault:my/path:MONGO_URI} for configuration connection.uri: The connection string is invalid. Connection strings must start with either 'mongodb://' or 'mongodb+srv://\nYou can al100 1295 100 439 100 856 173 337 0:00:02 0:00:02 --:--:-- 510rType}/config/validate`"}
|
// vault host, token and path redacted
This has been tested both in a local docker-compose environment with confluentinc/cp-kafka-connect:7.1.0. and real AWS MSK Connect environment
It look like even though the secret engine is invoked, the plugin incorectly then uses the secret invocation/interpolation syntax itself as the value, instead of using the retrieved secret
Could you please look into this?
If this is indeed a known bug or gap it renders the plugin unusable in a production env, hence the priority
Thank you,
Florin @ LeadIQ
Attachments
Issue Links
- is related to
-
KAFKA-361 MongoDB Source Connector - configuration validation runs before replacement when using a Config Provider
-
- Closed
-