Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-322

External secrets via ConfigProvider does not work

    • Type: Icon: Question Question
    • Resolution: Fixed
    • Priority: Icon: Critical - P2 Critical - P2
    • 1.10.0
    • Affects Version/s: None
    • Component/s: Configuration
    • None

      After packaging together with the jar a ConfigProvider (the standard way external secrets are added to Kafka Connect plugins) for the purposes of getting external secrets and configuring the config provider itself (in this case a VaultConfigProvider), e.g.

          "config.providers": "vault",
          "config.providers.vault.class": "com.github.jcustenborder.kafka.config.vault.VaultConfigProvider",
          "config.providers.vault.param.vault.address": "https://vault.domain.com",
          "config.providers.vault.param.vault.token": "atokenhere",
          "config.providers.vault.param.vault.login.by": "Token"
      

      // vault host, token and path redacted

      using / invoking the use of the secret in the plugin

       

      "connection.uri": "${vault:kv/my/path:MONGO_URI}",
      

       

      will result in the secret engine being created and invoked:

      INFO VaultConfigProviderConfig values: 
               vault.address = https://vault.domain.com
               vault.login.by = Token
               vault.max.retries = 5
               vault.namespace = 
               vault.prefix = 
               vault.retry.interval.ms = 2000
               vault.secret.minimum.ttl.ms = 1000
               vault.ssl.verify.enabled = true
               vault.token = [hidden]
        (com.github.jcustenborder.kafka.config.vault.VaultConfigProviderConfig:376)
      com.bettercloud.vault.Vault <init>
       INFO: Constructing a Vault instance with no provided Engine version, defaulting to version 2.
       INFO Authenticated to Vault as approle: path: auth/approle/login (com.github.jcustenborder.kafka.config.vault.AuthHandlers:131)
       INFO get() - path = 'my/path' keys = '[MONGO_URI]' (com.github.jcustenborder.kafka.config.vault.VaultConfigProvider:53)
       INFO VaultConfigProviderConfig values: 
               vault.address = https://vault.domain.com
               vault.login.by = Token
               vault.max.retries = 5
               vault.namespace = 
               vault.prefix = 
               vault.retry.interval.ms = 2000
               vault.secret.minimum.ttl.ms = 1000
               vault.ssl.verify.enabled = true
               vault.token = [hidden]
        (com.github.jcustenborder.kafka.config.vault.VaultConfigProviderConfig:376)
       Jul 22, 2022 6:32:41 AM com.bettercloud.vault.Vault <init>
       INFO: Constructing a Vault instance with no provided Engine version, defaulting to version 2.
       [2022-07-22 06:32:41,789] INFO Authenticated to Vault as approle: path: auth/approle/login (com.github.jcustenborder.kafka.config.vault.AuthHandlers:131)
       [2022-07-22 06:32:41,789] INFO get() - path = 'my/path' keys = '[MONGO_URI]' (com.github.jcustenborder.kafka.config.vault.VaultConfigProvider:53)
       [2022-07-22 06:32:42,009] INFO AbstractConfig values: 
        (org.apache.kafka.common.config.AbstractConfig:376)
      

       but then Kafka Connect, and maybe more specifically the mongo plugin throws this error:

      {"error_code":400,"message":"Connector configuration is invalid and contains the following 1 error(s):\nInvalid value ${vault:my/path:MONGO_URI} for configuration connection.uri: The connection string is invalid. Connection strings must start with either 'mongodb://' or 'mongodb+srv://\nYou can al100  1295  100   439  100   856    173    337  0:00:02  0:00:02 --:--:--   510rType}/config/validate`"}
      

      // vault host, token and path redacted

      This has been tested both in a local docker-compose environment with confluentinc/cp-kafka-connect:7.1.0. and real AWS MSK Connect environment

      It look like even though the secret engine is invoked, the plugin incorectly then uses the secret invocation/interpolation syntax itself as the value, instead of using the retrieved secret

      Could you please look into this?

      If this is indeed a known bug or gap it renders the plugin unusable in a production env, hence the priority

      Thank you,

      Florin @ LeadIQ

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            florin@leadiq.com Florin Bunău
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: