[KAFKA-42] how to handle non-JSON format messages when streaming data from Kafka to Mongo Created: 10/Jul/19  Updated: 11/Sep/19  Resolved: 10/Jul/19

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: 0.1
Fix Version/s: None

Type: Task Priority: Major - P3
Reporter: Vu Le Assignee: Ross Lawley
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Windows 7, CentOS Linux release 7.5.1804


Attachments: File MongoSinkConnector.properties     Text File log_error_when_stream_data_not_a_json_format.txt    

 Description   

Hi team,

I can stream data from Kafka to MongoDB with JSON messages

However, if I send a non-JSON format message the Connector died. Please see the log file for details.

My config file:

 

name=mongo-sink
topics=testconnector.class=com.mongodb.kafka.connect.MongoSinkConnector
tasks.max=1
key.ignore=true
 
# Specific global MongoDB Sink Connector configuration
connection.uri=mongodb://localhost:27017
database=test_kafka
collection=transaction
max.num.retries=3
retries.defer.timeout=5000
type.name=kafka-connect
 
key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false

I have 2 separated questions:

 

  1. how to ignore the message which is non-json format?
  1. how to defined a default-key for this kind of message (for example: abc -> {{ { "non-json": "abc" }

    }}

Thanks



 Comments   
Comment by Vu Le [ 10/Jul/19 ]

Hi Ross,

Thank for your response.
I am sorry because I didn't aware that where the problem come from, I thought it was from MongoDB Sink Connector. So, I could fix it by a proper configuration.

I raised this on stackoverflow already. But, so far, there is no answer.

I will reach to Kafka mailing list as you mentioned.

Thank you again.

Vu Le

Comment by Ross Lawley [ 10/Jul/19 ]

Hi leanhvu1989,

Thanks for the ticket. It seems its Kafka connect that is throwing the error before anything is handled by the MongoDB Sink Connector. As such there is nothing the connector itself can do.

Just for future reference the best place for questions regarding MongoDB usage or Kafka connector specifics is the mongodb-user mailinglist or stackoverflow as you will reach a broader audience there. If your business requires an answer from MongoDB within a time frame then we do offer production support.

In this case I think using Kafka's own mailing list may be the be best way to get insight into how best to handle this scenario.

All the best,

Ross

Generated at Thu Feb 08 09:05:26 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.