Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-386

Kafka mongo sink. Create _id from key message

    XMLWordPrintableJSON

Details

    • Icon: Question Question
    • Resolution: Works as Designed
    • Icon: Minor - P4 Minor - P4
    • None
    • None
    • None
    • None

    Description

      Hello,

      I have the following kafka message: key content is {{

      {orderid: 123}

      }} and the message value as {{{}

      {name: "def", company: "abc"}

      {}}}. I want to insert in mongodb the following document: {_id: 123, name: "def", company: "abc"} where _id takes the value of orderid from the key message. I have tried the following config and it looks ok except that the _id is being an object like this: _id:{orderid: 123}{}. Any idead how we can achieve this? I have the following config:

      name = MongoSinkConnectorConnector_1
      connector.class = com.mongodb.kafka.connect.MongoSinkConnector
      topics = topicA
       
      key.converter = org.apache.kafka.connect.storage.StringConverter
      key.converter.schemas.enable = false
       
      value.converter = org.apache.kafka.connect.json.JsonConverter
      value.converter.schemas.enable = false
       
      connection.uri = mongodb://localhost:27017
      database = foo
      collection = mycollection
       
      # Use a single message as a document
      document.id.strategy=com.mongodb.kafka.connect.sink.processor.id.strategy.PartialKeyStrategy
       
      # Post processing of the sink documents
      post.processor.chain=com.mongodb.kafka.connect.sink.processor.DocumentIdAdder
       
      # Document ID strategy using the Partial Key Strategy
      document.id.strategy.overwrite.existing=true
       
      # Provide a custom setting for which key values to consider for the _id
      document.id.strategy.partial.key.projection.list=orderid,customerid
      document.id.strategy.partial.key.projection.type=AllowList 

      Thank you!

      Ralph

       

      Attachments

        Activity

          People

            robert.walters@mongodb.com Robert Walters
            ralph.sawaya@mongodb.com Ralph Sawaya
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: