[KAFKA-371] MongoDB Kafka Connect Connectors - config values provided by ConfigProvider does not replace placeholders in configuration Created: 15/May/23  Updated: 16/Nov/23  Resolved: 27/Jul/23

Status: Closed
Project: Kafka Connector
Component/s: Configuration, Sink, Source
Affects Version/s: 1.10.0
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Lincoln D'Mello Assignee: Ross Lawley
Resolution: Gone away Votes: 0
Labels: Configuration
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Problem/Incident
is caused by KAFKA-361 MongoDB Source Connector - configurat... Closed

 Description   

Environment:

Kafka Connect cluster v7.30 running on AWS EKS with configured IRSA service account for the EKS nodes with the correct IAM permissions to retrieve secrets from AWS Secrets Manager

ConfigProvider being used: SecretsManagerConfigProvider - https://github.com/jcustenborder/kafka-config-provider-aws

Mongo Source Connector being used: https://www.confluent.io/hub/mongodb/kafka-connect-mongodb

Problem:
I have tested the latest version of the connectors with the above config provider and the problem reported on ticket KAFKA-361 still persists.

    However, this time around when I validate the connector configuration with Kafka Connect via the REST validation endpoint, I can see that, in the JSON validation error that is returned from Kakfa Connect, for the 'connection.uri'  config property, the 'value.value' JSON property has the correct uri value that the config provider has retrieved from AWS but somehow the retrieved value has still not replaced the placeholder 

 

"${secretmanager:dev/db/mongo/test/connection:uri}"

 

in the actual configuration that Kafka Connect is trying to use to configure the connector instance.
   When I try to create the connector using the above config, I just get an error saying that the 'connection.uri' has an invalid value, with the placeholder above shown to be the invalid value.



 Comments   
Comment by Ed Berezitsky [ 16/Nov/23 ]

danuta.stawiarz@gmail.com ,

if you define config provider in the env vars, why would you redefine it again in a connector config? Can you keep env vars (as mentioned in the last comment), but remove it from connector config?

Fixed sample:

connector.class=com.mongodb.kafka.connect.MongoSinkConnector
database=test
# config.providers.secretsManager.class=io.confluent.csid.config.provider.aws.SecretsManagerConfigProvider
# config.providers=secretsManager
topics=test-topic
connection.uri= ${secretsManager:my-secret:connectionstring} 
collection=${secretsManager:my-secret:collection} 

Comment by Danuta Stawiarz [ 08/Nov/23 ]

Hi bdesert@gmail.com ,

thanks a lot for answering. Sorry for not being precise in my previous description! 

Our kafka connect is hosted on ECS instance and  we pass there environmental variables: 

      {
        "name": "CONNECT_CONFIG_PROVIDERS",
        "value": "secretsManager"
      },
      {
        "name": "CONNECT_CONFIG_PROVIDERS_SECRETSMANAGER_CLASS",
        "value": "io.confluent.csid.config.provider.aws.SecretsManagerConfigProvider"
      }, 

we didn't find worker.properties file to add what you mentioned. 

 

Comment by Ed Berezitsky [ 08/Nov/23 ]

hi danuta.stawiarz@gmail.com ,

To solve this issue, you need to define a config providers in the worker properties, not in connector. Move the following lines into the worker.properties file:

 

config.providers.secretsManager.class=io.confluent.csid.config.provider.aws.SecretsManagerConfigProvider
config.providers=secretsManager

 

 

Comment by Danuta Stawiarz [ 08/Nov/23 ]

Hello, 

We are experiencing similar problem. We use similar stack provided by the ticket author: 
– Mongo Source Connector  1.11.0 version 
– ConfigProvider: SecretsManagerConfigProvider -https://www.confluent.io/hub/confluentinc/csid-secrets-provider-aws/

As suggested, worker has been updated with the config: 

CONNECT_CONFIG_PROVIDERS = "secretsManager"
CONNECT_CONFIG_PROVIDERS_SECRETSMANAGER_CLASS = "io.confluent.csid.config.provider.aws.SecretsManagerConfigProvider"
CONNECT_CONFIG_PROVIDERS_SECRETSMANAGER_PARAM_AWS_REGION= "eu-west-1"

The problem appears to exist only in connection.uri field. When assigning secret value to any other field, like collection, secret values are properly retrieved. In case of connection.uri everything is treated as String.  We have tried with the following variants:

connection.uri = ${secretsManager:my-secret:fullconnection}

connection.uri = mongodb://${secretsManager:my-secret:username}:$${secretsManager:my-secret:password}@${secretsManager:gmy-secret:connection.uri}

connection.uri = mongodb%3A%2F%2F%24%7BsecretsManager%3Amy-secret%3Ausername%7D%3A%24%24%7BsecretsManager%3Amy-secret%3Apassword%7D%40%24%7BsecretsManager%3Agmy-secret%3Aconnection.uri%7D

In logs and errors we could see that connection.uri value is  never resolved and always treated as string. Example connector config: 

connector.class=com.mongodb.kafka.connect.MongoSinkConnector
database=test
config.providers.secretsManager.class=io.confluent.csid.config.provider.aws.SecretsManagerConfigProvider
topics=test-topic
connection.uri= ${secretsManager:my-secret:connectionstring} 
config.providers=secretsManager
collection=${secretsManager:my-secret:collection} 

All the best,

Danuta

Comment by PM Bot [ 27/Jul/23 ]

There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to reopen this ticket if you're still experiencing the issue, and add a comment if you're able to provide more information.

Comment by Bryan Kolterman [ 18/Jul/23 ]

Hi ross@mongodb.com
Thanks for the update and for the investigation. I will pass this note along to the customer. 

-Bryan

Comment by PM Bot [ 10/Jul/23 ]

There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to reopen this ticket if you're still experiencing the issue, and add a comment if you're able to provide more information.

Comment by Ross Lawley [ 26/Jun/23 ]

HI lincoln42@hotmail.com,

Thanks for the update - just wanted to double check if you have logged this issue on our MongoDB support portal, located at support.mongodb.com?

However, when the same validated configuration is used to try and create a connector instance on Kafka Connect, it fails with the same error encountered above which says that the 'connection.uri' in invalid, where the connection.uri is masked as mentioned above.

To determine the cause of the error on Kafka connect could you provide more detail on the bug? Either a full stacktrace or a test case to reproduce the error. Ideally, a minimal reproducible example would help as I could replicate the bug and use it as a test case for the fix.

All the best,

Ross Lawley

Comment by Lincoln D'Mello [ 24/Jun/23 ]

bryan.kolterman@mongodb.com, bdesert@gmail.com, ross@mongodb.com I have re-tested the connector with the config provider above along with the suggested changes to place the config provider configuration in the worker's properties file. Now, with those changes in place, the connector's configuration passes validation with no reported errors.
   However, when the same validated configuration is used to try and create a connector instance on Kafka Connect, it fails with the same error encountered above which says that the 'connection.uri' in invalid, where the connection.uri is masked as mentioned above.
  Please re-open this ticket as the issue is not resolved fully.

Comment by PM Bot [ 21/Jun/23 ]

There hasn't been any recent activity on this ticket, so we're resolving it. Thanks for reaching out! Please feel free to reopen this ticket if you're still experiencing the issue, and add a comment if you're able to provide more information.

Comment by Ed Berezitsky [ 14/Jun/23 ]

bryan.kolterman@mongodb.com sent you an email, taking offline until we get get clear picture of the root cause.

Comment by PM Bot [ 13/Jun/23 ]

Hi lincoln42@hotmail.com! KAFKA-371 is awaiting your response.

If this is still an issue for you, please open Jira to review the latest status and provide your feedback. Thanks!

Comment by Ross Lawley [ 06/Jun/23 ]

Hi lincoln42@hotmail.com,

I'll leave this ticket open until you can confirm I've identified the issue or not. Please let me know how you get on.

All the best,

Ross

Comment by Ross Lawley [ 06/Jun/23 ]

Hi lincoln42@hotmail.com,

Please ensure that the Kafka workers are configured correctly to use a ConfigProvider as its the worker configuration and not the connector configuration that has to declare the provider:

See: kafka-config-provider-aws documentation

Worker Configuration
Add the following configuration to your Connect worker config file to enable this ConfigProvider to be used by connectors.

config.providers.secretManager.param.secret.prefix=staging
config.providers.secretManager.class=com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider
config.providers=secretManager
config.providers.secretManager.param.aws.region=us-west-2

Connector Configuration
Add the following configurations to your connector configuration when you would like to retrieve a value from the config provider

{
  "username" : "${secretManager:secret/test/some/connector:username}",
  "password" : "${secretManager:secret/test/some/connector:password}"
}

Hope that helps,

Ross

Comment by Bryan Kolterman [ 02/Jun/23 ]

Hi bdesert@gmail.com
Thanks for your help with this. My name is Bryan Kolterman and I am the Customer Success Manager at Mongo covering Fitch Ratings. Lincoln replied back via email about the requested details. His reply is below: 

I believe it's best to set up a zoom call. Are you available this afternoon to continue the conversation on zoom? 

Hello Bryan,
                        I have chosen to reply to the request for information via this email rather than on the ticket publicly, since the configs and logs contain information specific to Fitch internal systems. I have attached the information requested by the engineer to this message as a Zip file. If more clarity is needed we could schedule a Zoom call between myself and the engineer.

     We are running Confluent Platform’s Kafka Connect cluster on AWS EKS.
    Please note, in the response for the validation request I have overwritten our MongoDB  password but rest assured it shows that the config provider is picking up the secret value from AWS Secrets Manager using the IAM Role assigned to the IRSA service account used by the POD’s K8 deployment.

Comment by Ed Berezitsky [ 01/Jun/23 ]

lincoln42@hotmail.com , can you please provide the following details:

  1. worker's properties
  2. connector's properties
  3. recursive tree view of `plugin.path` directory
  4. Connector's logs

 

Comment by PM Bot [ 15/May/23 ]

Hi lincoln42@hotmail.com, thank you for reporting this issue! The team will look into it and get back to you soon.

Generated at Thu Feb 08 09:06:13 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.