Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-42

how to handle non-JSON format messages when streaming data from Kafka to Mongo

    XMLWordPrintableJSON

Details

    • Icon: Task Task
    • Resolution: Done
    • Icon: Major - P3 Major - P3
    • None
    • 0.1
    • None
    • None
    • Windows 7, CentOS Linux release 7.5.1804

    Description

      Hi team,

      I can stream data from Kafka to MongoDB with JSON messages

      However, if I send a non-JSON format message the Connector died. Please see the log file for details.

      My config file:

       

      name=mongo-sink
      topics=testconnector.class=com.mongodb.kafka.connect.MongoSinkConnector
      tasks.max=1
      key.ignore=true
       
      # Specific global MongoDB Sink Connector configuration
      connection.uri=mongodb://localhost:27017
      database=test_kafka
      collection=transaction
      max.num.retries=3
      retries.defer.timeout=5000
      type.name=kafka-connect
       
      key.converter=org.apache.kafka.connect.json.JsonConverter
      key.converter.schemas.enable=false
      value.converter=org.apache.kafka.connect.json.JsonConverter
      value.converter.schemas.enable=false
      
      

      I have 2 separated questions:

       

      1. how to ignore the message which is non-json format?
      1. how to defined a default-key for this kind of message (for example: abc -> {{ { "non-json": "abc" }

        }}

      Thanks

      Attachments

        Activity

          People

            ross@mongodb.com Ross Lawley
            leanhvu1989 Vu Le
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: