[KAFKA-144] key.projection.type and key.projection.list not respecting Created: 21/Aug/20  Updated: 27/Oct/23  Resolved: 21/Aug/20

Status: Closed
Project: Kafka Connector
Component/s: Sink
Affects Version/s: 1.2.0
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Rajaramesh Yaramati Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Kafka 2.6



 Description   

I am testing key.projection.type & key.projection.list options and looks like it is not respecting. On the destination database, I am receiving a field I am adding to the blocklist. Below sink configuration, I am using. Can you please let me know is there any known issue or I am doing something wrong? 

 

curl -X POST -H "Accept:application/json" -H "Content-Type: application/json" localhost:9083/connectors/ --data '{

"name":"mongo-sink-norevisions-mongos",

"config": {

"topics":"oplog.oz_mongo.oz_next.assets",

"connector.class":"com.mongodb.kafka.connect.MongoSinkConnector",

"tasks.max":"1",

"key.converter":"org.apache.kafka.connect.json.JsonConverter",

"key.converter.schemas.enable":"false",

"value.converter":"org.apache.kafka.connect.json.JsonConverter",

"value.converter.schemas.enable":"false",

"connection.uri":"mongodb://xxx.xxx.xxx.xxx:27017",

"database":"poc_oz_next",

"collection":"poc_assets_norevision",

"max.num.retries":"3",

"retries.defer.timeout":"5000",

"session.timeout.ms":"25000",

"delete.on.null.values":"true",

"key.projection.type":"blocklist",

"key.projection.list":"revisions,revision_ids"}

}'

 

Thanks,

Rajaramesh



 Comments   
Comment by Ross Lawley [ 21/Aug/20 ]

Hi yaramati@adobe.com,

You will also need to configure the PostProcessor, which will use those configurations and apply them - see: https://docs.mongodb.com/kafka-connector/current/kafka-sink-postprocessors/#blocklist-allowlist-projector

For future reference, this sounds like a support issue, I wanted to give you some resources to get this questioned answered more quickly:

  • our MongoDB support portal, located at support.mongodb.com
  • our MongoDB community portal, located here
  • If you are an Atlas customer, there is free support offered 24/7 in the lower right hand corner of the UI.

Just in case you have already opened a support case and are not receiving sufficient help, please let me know and I can facilitate escalating your issue.

Thank you!

Ross

Generated at Thu Feb 08 09:05:41 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.