how to handle non-JSON format messages when streaming data from Kafka to Mongo

XMLWordPrintableJSON

    • Type: Task
    • Resolution: Done
    • Priority: Major - P3
    • None
    • Affects Version/s: 0.1
    • Component/s: None
    • None
    • Environment:
      Windows 7, CentOS Linux release 7.5.1804
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Hi team,

      I can stream data from Kafka to MongoDB with JSON messages

      However, if I send a non-JSON format message the Connector died. Please see the log file for details.

      My config file:

       

      name=mongo-sink
topics=testconnector.class=com.mongodb.kafka.connect.MongoSinkConnector
tasks.max=1
key.ignore=true

# Specific global MongoDB Sink Connector configuration
connection.uri=mongodb://localhost:27017
database=test_kafka
collection=transaction
max.num.retries=3
retries.defer.timeout=5000
type.name=kafka-connect

key.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable=false

      I have 2 separated questions:

       

      1. how to ignore the message which is non-json format?
      1. how to defined a default-key for this kind of message (for example: abc -> {{ { "non-json": "abc" }

        }}

      Thanks

        1. log_error_when_stream_data_not_a_json_format.txt
          5 kB
        2. MongoSinkConnector.properties
          0.6 kB

            Assignee:
            Ross Lawley
            Reporter:
            Vu Le
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: