[KAFKA-48] Can't set custom message key Created: 23/Jul/19  Updated: 27/Oct/23  Resolved: 25/Jul/19

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Dionis Stepanidis Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Depends
depends on KAFKA-40 Support schema for Source connector Closed

 Description   

I try to use mongo document id as message key. Following this examples https://docs.confluent.io/current/connect/transforms/valuetokey.html#valuetokey i specify settings in MongoSourceConnector.properties but connector raise exception:

 

Only Struct objects supported for [copying fields from value to key], found: java.lang.String

 



 Comments   
Comment by Ross Lawley [ 25/Jul/19 ]

I'm closing as Works as Designed - only because Kafka and the connector are working as they currently are supposed to.

Please follow / watch KAFKA-40 for updates about supporting schema'd data from the source connector, which will then allow you to use Kafka's value to key SMT.

Ross

Comment by Dionis Stepanidis [ 24/Jul/19 ]

If you are using the Mongo Kafka Connector as the source and want to apply this SMT then you will have to wait until KAFKA-40 to support schema'd data from the source.

I will wait that feature. Thanks for help Ross

Comment by Ross Lawley [ 24/Jul/19 ]

Hi industrial.lang@gmail.com,

It depends how you are using the SMT feature. If you are applying it to data from a source connector then it will be applied before the hits kafka. If its for a sink connector then its applied to the data from kafka and before the data hits the sink connector.

What is the source of the data? Does it support producing data of different formats? How have you configured the value serializer?

If you are using the Mongo Kafka Connector as the source and want to apply this SMT then you will have to wait until KAFKA-40 to support schema'd data from the source.

I hope that helps,

Ross

Comment by Dionis Stepanidis [ 24/Jul/19 ]

Hi Ross, thanks for replay.

Maybe i don't fully understand data flow from mongo to kafka through mongo connector.
I imagine that process as flow of json documents from mongo to connector and then to kafka. Kafka before add this document to topic can make some transormations(Kafka SMT feature).
I try to do some transformations but kafka says "Only Struct objects supported for [copying fields from value to key], found: java.lang.String".

From this I concluded that from the connector to the Kafka not the structure (json) is passed, but the string.

Comment by Ross Lawley [ 24/Jul/19 ]

Hi industrial.lang@gmail.com,

Thanks for the ticket. Just to let you know the value to key mechanism is handled by Kafka connect and not the MongoDB Connector. From the exception it sounds like your data in the topic is a string and but the value to key mechanism requires it to be a Struct (so Json or Avro).

Ross

Generated at Thu Feb 08 09:05:27 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.