[KAFKA-81] Schema registry credentials not taken from kafka connect Created: 30/Jan/20  Updated: 17/Jun/20  Resolved: 17/Jun/20

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: 1.0
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Ludovic Dussart Assignee: Ross Lawley
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

kafka connect 5.3.1



 Description   

hi,

 

We are using Mongo Sink with the schema registry.

The schema registry authentication is handle by the kafka connect instance thanks to the related keys :

 

 
key.converter.schema.registry.url=xxx
key.converter.basic.auth.credentials.source=USER_INFO
key.converter.basic.auth.user.info=xxx
 
value.converter.schema.registry.url=xxx
value.converter.basic.auth.credentials.source=USER_INFO
value.converter.basic.auth.user.info=xxx
 

 

 

When deploying a mongo-sink configuration, if we not add thoses keys, connect throws an Unauthorized exception when trying to get key and value schemas from registry.

With others connectors like sftp connect (provided by Confluent), this configuration is induced from kafka-connect configuration listed below.

 

It will be nice that the Mongo connector hava the same behavior.
 



 Comments   
Comment by Ross Lawley [ 05/Feb/20 ]

Hi ludovic.dussart.pro@gmail.com,

Thanks for the clarification. This is strange and looks to me to be a Kafka Connect bug. As mentioned previously the MongoDB Connector does nothing with any configurations outside its remit.

From the documentation you've linked it looks like the workers configuration should be used and if specified overrides the connector configurations. So from that it should work and as it doesn't this sounds like a Kafka Connect issue.

Ross

Comment by Ludovic Dussart [ 04/Feb/20 ]

hi @Ross,

 

According to the Kafka Connect documentation :

Confluent Platform first looks for converter configurations in the connector. If none are found there, the settings in the Connect worker configs are used. Therefore, you have options as to where to specify these properties, which affects how they are inherited among the worker and connectors.

  • Specify all (converters and Schema Registry URL prefixed properties) individually in each connector configuration.
  • Specify all properties only in the worker configuration, in which case the same settings will be used by all connectors.
  • Specify all properties in the worker configuration, and specify all overrides in the connectors.

The two last items was apparently not managed by the Mongo Connector because i need to explicitly declared value|key.converter.basic.auth.credentials.source and value|key.converter.basic.auth.user.info even if my kafka connect worker have those configurations
 

Comment by Ross Lawley [ 04/Feb/20 ]

Hi ludovic.dussart.pro@gmail.com,

Thanks for the ticket, I'm not sure I follow I think the example config was missing from the description.

The MongoDB Connector itself does not handle or use the (key|value.converter) namespace, rather it is purely used by the connect framework. See the Using Kafka Connect with Schema Registry documentation for more information about how to configure a connector with the schema registry.

All the best,

Ross

Generated at Thu Feb 08 09:05:31 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.