[KAFKA-322] External secrets via ConfigProvider does not work Created: 22/Jul/22  Updated: 28/Oct/23  Resolved: 02/Mar/23

Status: Closed
Project: Kafka Connector
Component/s: Configuration
Affects Version/s: None
Fix Version/s: 1.10.0

Type: Question Priority: Critical - P2
Reporter: Florin Bunău Assignee: Ross Lawley
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
is related to KAFKA-361 MongoDB Source Connector - configurat... Closed
Quarter: FY24Q1
Case:

 Description   

After packaging together with the jar a ConfigProvider (the standard way external secrets are added to Kafka Connect plugins) for the purposes of getting external secrets and configuring the config provider itself (in this case a VaultConfigProvider), e.g.

    "config.providers": "vault",
    "config.providers.vault.class": "com.github.jcustenborder.kafka.config.vault.VaultConfigProvider",
    "config.providers.vault.param.vault.address": "https://vault.domain.com",
    "config.providers.vault.param.vault.token": "atokenhere",
    "config.providers.vault.param.vault.login.by": "Token"

// vault host, token and path redacted

using / invoking the use of the secret in the plugin

 

"connection.uri": "${vault:kv/my/path:MONGO_URI}",

 

will result in the secret engine being created and invoked:

INFO VaultConfigProviderConfig values: 
         vault.address = https://vault.domain.com
         vault.login.by = Token
         vault.max.retries = 5
         vault.namespace = 
         vault.prefix = 
         vault.retry.interval.ms = 2000
         vault.secret.minimum.ttl.ms = 1000
         vault.ssl.verify.enabled = true
         vault.token = [hidden]
  (com.github.jcustenborder.kafka.config.vault.VaultConfigProviderConfig:376)
com.bettercloud.vault.Vault <init>
 INFO: Constructing a Vault instance with no provided Engine version, defaulting to version 2.
 INFO Authenticated to Vault as approle: path: auth/approle/login (com.github.jcustenborder.kafka.config.vault.AuthHandlers:131)
 INFO get() - path = 'my/path' keys = '[MONGO_URI]' (com.github.jcustenborder.kafka.config.vault.VaultConfigProvider:53)
 INFO VaultConfigProviderConfig values: 
         vault.address = https://vault.domain.com
         vault.login.by = Token
         vault.max.retries = 5
         vault.namespace = 
         vault.prefix = 
         vault.retry.interval.ms = 2000
         vault.secret.minimum.ttl.ms = 1000
         vault.ssl.verify.enabled = true
         vault.token = [hidden]
  (com.github.jcustenborder.kafka.config.vault.VaultConfigProviderConfig:376)
 Jul 22, 2022 6:32:41 AM com.bettercloud.vault.Vault <init>
 INFO: Constructing a Vault instance with no provided Engine version, defaulting to version 2.
 [2022-07-22 06:32:41,789] INFO Authenticated to Vault as approle: path: auth/approle/login (com.github.jcustenborder.kafka.config.vault.AuthHandlers:131)
 [2022-07-22 06:32:41,789] INFO get() - path = 'my/path' keys = '[MONGO_URI]' (com.github.jcustenborder.kafka.config.vault.VaultConfigProvider:53)
 [2022-07-22 06:32:42,009] INFO AbstractConfig values: 
  (org.apache.kafka.common.config.AbstractConfig:376)

 but then Kafka Connect, and maybe more specifically the mongo plugin throws this error:

{"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nInvalid value ${vault:my/path:MONGO_URI} for configuration connection.uri: The connection string is invalid. Connection strings must start with either 'mongodb://' or 'mongodb+srv://\nYou can al100  1295  100   439  100   856    173    337  0:00:02  0:00:02 --:--:--   510rType}/config/validate`"}

// vault host, token and path redacted

This has been tested both in a local docker-compose environment with confluentinc/cp-kafka-connect:7.1.0. and real AWS MSK Connect environment

It look like even though the secret engine is invoked, the plugin incorectly then uses the secret invocation/interpolation syntax itself as the value, instead of using the retrieved secret

Could you please look into this?

If this is indeed a known bug or gap it renders the plugin unusable in a production env, hence the priority

Thank you,

Florin @ LeadIQ



 Comments   
Comment by Ross Lawley [ 02/Mar/23 ]

Thanks bdesert@gmail.com for the PR.

Comment by Githook User [ 02/Mar/23 ]

Author:

{'name': 'Ed B', 'email': 'bdesert@gmail.com', 'username': 'bdesert'}

Message: Ensure configs are parsed and evaluated (#133)

When using config providers the `connection.uri` might not be processed. The reason is that validate function of either Sink or Source Connector, the config values are VALIDATED, but not PARSED.

This fix ensures the values are parsed then validated

KAFKA-322
Branch: master
https://github.com/mongodb/mongo-kafka/commit/121aa9d55a1b597a19250eaed9248f9b610a8da4

Comment by Ed Berezitsky [ 01/Mar/23 ]

@ross@mongodb.com ,

I have a fix for this issue, if you have some cycles to review.

 

Comment by Ross Lawley [ 26/Jan/23 ]

Hi bdesert@gmail.com,

I have moved this into the backlog and will look to get it scheduled . In the meantime if you are able to provide a PR just for your validation fix, that would be great and help speed up the development.

All the best,

Ross

Comment by Ed Berezitsky [ 19/Jan/23 ]

@Ross Lawley,

it doesn't work for me, I use multiple different config providers, and `connection.uri` is not being processed.

The reason is that validate function of either Sink or Source Connector, the config values are VALIDATED, but not PARSED.

For example, Sink Connector:

Config is created from the validator.

Since this config is used for validation of the connection, the `connection.uri` is used as is, without parsing.

To work it around, I override a value or `connection.uri` in `connectorConfigs` with a parsed values from sinkConfig (which is parsed config).

To properly fix it, validation functionality should utilize `sinkConfig` instead of `config` (or in addition to that).

 

 

Comment by Florin Bunău [ 29/Jul/22 ]

I can confirm the plugin works both with Vault and AWS Secret Manager ConfigProviders

Comment by Florin Bunău [ 29/Jul/22 ]

Thank you @Ross Lawley 

Indeed this was the problem. I don't know how I misread the initial docs, I was looking at this for a couple days, couldn't figure out what was wrong. Upon careful inspection even AWS docs state a connection to the worker config. Sorry for the false alarm, this was very helpful

Comment by Ross Lawley [ 27/Jul/22 ]

Hi florin@leadiq.com,

ConfigProviders should be opaque to the connectors - the connectors know nothing of them and the configurations provided to the connector should have already been transformed by the config provider.
So this looks to be a set up issue - similar to https://stackoverflow.com/questions/69076149/mongosinkconnector-resolving-secret-throws-validation-exception

Please ensure that the config provider is configured on the Kafka workers / executors level and not along side the connector.

Ross

Comment by Florin Bunău [ 26/Jul/22 ]

Hi! an update on this

 

I also tried the AWS Secret Manager on a real AWS MSK Connect deployment, as described by the official documentation https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-config-provider.html packaging up the Mongo plugin together with https://github.com/jcustenborder/kafka-config-provider-aws ( note needs Guava 31.1-jre packaged up too https://github.com/google/guava/releases for it to work) 

 

and it gives out the same error:

Message: The connector configuration is invalid. Message: Connector configuration is invalid and contains the following 1 error(s): Invalid value ${secretManager:dev/kafka-connector:MONGO_URI} for configuration connection.uri: The connection string is invalid. Connection strings must start with either 'mongodb://' or 'mongodb+srv:// You can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`

 

After the secret is retrieved, the mongo plugin just continues to use the string that invokes the secret instead of the actual retrieved secret value

 

Generated at Thu Feb 08 09:06:06 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.