[KAFKA-369] External secrets via ConfigProvider does not work in mongodb source connector Created: 12/May/23  Updated: 27/Oct/23  Resolved: 06/Jun/23

Status: Closed
Project: Kafka Connector
Component/s: Configuration
Affects Version/s: 1.10.0
Fix Version/s: None

Type: Question Priority: Unknown
Reporter: 재민 심 Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Environment

On-Prem

  • Confluent Kafka 7.3.2 (Community)
  • Confluent Connect 7.3.2 (Community)

Problem:

I am trying to encrypt "connection.uri" configuration of mongodb source connector through Vault ConfigProvider.

 

My configuration is here

{
        "config.providers.vault.param.vault.namespace": "jaeshim"
        "connector.class": "com.mongodb.kafka.connect.MongoSourceConnector",
        "publish.full.document.only": "true",
        "output.format.value": "json",
        "config.providers": "vault",
        "output.format.key": "json",
        "config.providers.vault.param.vault.login.by": "Token",
        "config.providers.vault.class": "com.github.jcustenborder.kafka.config.vault.VaultConfigProvider",
        "topic.prefix": "jaeshim_prefix",
        "database": "jaeshim",
        "config.providers.vault.param.vault.token": "jaeshim_token",
        "topic.separator": "_",
        "config.providers.vault.param.vault.address": "https://vault-url:8200",
        "connection.uri": "${vault:kconnect-kv/jaeshim/mongodb:connection-uri}"
        "name": "MongoSourceConnector_Jaeshim",        
        "value.converter": "org.apache.kafka.connect.storage.StringConverter",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter"
 } 

but I got the error

{    "error_code": 400,    "message": "Connector configuration is invalid and contains the following 1 error(s):\nInvalid value ${vault:kconnect-kv/jaeshim/mongodb:connection-uri}, for configuration connection.uri: The connection string is invalid. Connection strings must start with either 'mongodb://' or 'mongodb+srv://\nYou can also find the above list of errors at the endpoint `/connector-plugins/{connectorType}/config/validate`"} 

and I found similar issue. We have same problem. 

https://jira.mongodb.org/browse/KAFKA-322 

https://github.com/mongodb/mongo-kafka/commit/121aa9d55a1b597a19250eaed9248f9b610a8da4 

 

According to the report, It is known issue and fixed at version 1.10.

We are using version 1.10 but we still get the same error. 

 

Could this issue be checked?



 Comments   
Comment by Ross Lawley [ 06/Jun/23 ]

Hi sjm7267@gmail.com,

Thanks for updating the ticket with the cause.

For future readers please ensure that the Kafka workers are configured correctly to use a ConfigProvider as its the worker configuration and not the connector configuration that has to declare the provider.

All the best,

Ross

Comment by 재민 심 [ 18/May/23 ]

hello
This problem has been solved by setting "config.provider" at the Worker level by referring to the contents of the existing JIRA.

 

https://jira.mongodb.org/browse/KAFKA-322

https://stackoverflow.com/questions/69076149/mongosinkconnector-resolving-secret-throws-validation-exception 

https://docs.confluent.io/platform/current/installation/configuration/connect/index.html#config-providers 

Comment by 재민 심 [ 12/May/23 ]

Additionaly we are use vault token with ENVIRONMENT_VARIABLE. 

Other connectors are working fine. 

 

And here is our souce connector version

 

#request
http://connect:8083/connector-plugins
 
#response
    {        "class""com.mongodb.kafka.connect.MongoSourceConnector",        "type""source",        "version""1.10.0"    }, 

Comment by PM Bot [ 12/May/23 ]

Hi sjm7267@gmail.com, thank you for reporting this issue! The team will look into it and get back to you soon.

Generated at Thu Feb 08 09:06:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.