Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-361

MongoDB Source Connector - configuration validation runs before replacement when using a Config Provider

    • Type: Icon: Question Question
    • Resolution: Duplicate
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.9.1
    • Component/s: Source
    • Labels:
      None

      Environment:
      Confluent Stack running using Docker Compose (Kafka, KafkaConnect, Zookeeper, etc.) Version: 7.3.0
      ConfigProvider being used: SecretsManagerConfigProvider - https://github.com/jcustenborder/kafka-config-provider-aws
      Mongo Source Connector being used: https://www.confluent.io/hub/mongodb/kafka-connect-mongodb

      Problem:
      When using the Kafka Connect REST API to either validate or create a new MongoSourceConnector instance when using the above configuration provider to replace the values of secret configuration items in the configuration as below. I get the error - Invalid value "${secretmanager:dev/db/mongo/test/connection:uri}" for configuration connection.uri

      Additionally, I am using an AWS Profile to provide the AWS credentials and I can confirm that I can fetch the secrets from AWS Secrets Manager just fine via the AWS CLI for the profile being used. The AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE environment variables have been set on the Kafka Connect container as well. Also, the Mongo Source connector works just fine when not using a config provider and with a directly supplied connection.url value.

       {
        "connector.class" : "com.mongodb.kafka.connect.MongoSourceConnector",
        "config.providers" : "secretmanager",
        "config.providers.secretmanager.class"  : "com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider",
        "config.providers.secretmanager.param.aws.region" : "us-east-1",
        "tasks.max" : 1,
        "key.converter" : "io.confluent.connect.avro.AvroConverter",
        "value.converter" : "io.confluent.connect.avro.AvroConverter",
        "errors.tolerance" : "all",
        "errors.log.enable": "true",
        "errors.log.include.messages" : "false",
        "errors.deadletterqueue.context.headers.enable": "true",
        "errors.deadletterqueue.topic.name" : "dlq",
        "connection.uri" : "${secretmanager:dev/db/mongo/test/connection:uri}"
      }
      

      Could this issue be linked to, or a regression of, #737 ?

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            lincoln42@hotmail.com Lincoln D'Mello
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: