[KAFKA-386] Kafka mongo sink. Create _id from key message Created: 20/Sep/23  Updated: 27/Oct/23  Resolved: 10/Oct/23

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Minor - P4
Reporter: Ralph Sawaya Assignee: Robert Walters
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Hello,

I have the following kafka message: key content is {{

{orderid: 123}

}} and the message value as {{{}

{name: "def", company: "abc"}

{}}}. I want to insert in mongodb the following document: {_id: 123, name: "def", company: "abc"} where _id takes the value of orderid from the key message. I have tried the following config and it looks ok except that the _id is being an object like this: _id:{orderid: 123}{}. Any idead how we can achieve this? I have the following config:

name = MongoSinkConnectorConnector_1
connector.class = com.mongodb.kafka.connect.MongoSinkConnector
topics = topicA
 
key.converter = org.apache.kafka.connect.storage.StringConverter
key.converter.schemas.enable = false
 
value.converter = org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable = false
 
connection.uri = mongodb://localhost:27017
database = foo
collection = mycollection
 
# Use a single message as a document
document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy.PartialKeyStrategy
 
# Post processing of the sink documents
post.processor.chain=com.mongodb.kafka.connect.sink.processor.DocumentIdAdder
 
# Document ID strategy using the Partial Key Strategy
document.id.strategy.overwrite.existing=true
 
# Provide a custom setting for which key values to consider for the _id
document.id.strategy.partial.key.projection.list=orderid,customerid
document.id.strategy.partial.key.projection.type=AllowList 

Thank you!

Ralph

 



 Comments   
Comment by Ralph Sawaya [ 26/Sep/23 ]

Thanks Robert. This works well if the key message contains the _id field such as {_id: 123} but in my case the key message can contain either orderid or itemid. I will check with the customer if he is willing to make a change on the message structure. Otherwise I believe, after much investigation, we are just left with custom transformation to achieve the desired result. It is anyway ok for the customer to keep things as they are but it would have been better if there was a staighforward solution so that _id contains directly the value instead of the object with the value.

Comment by Robert Walters [ 25/Sep/23 ]

Consider using the document.id.strategy field as defined below:

https://www.mongodb.com/docs/kafka-connector/current/sink-connector/fundamentals/post-processors/#configure-the-document-id-adder-post-processor

 

document.id.strategy": "com.mongodb.kafka.connect.sink.processor.id.strategy.ProvidedInKeyStrategy"

 

 

Generated at Thu Feb 08 09:06:15 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.