Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-75

New id strategy that doesn't reuse configuration parameters.

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major - P3
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
      None
    • Documentation Changes:
      Needed
    • Documentation Changes Summary:
      Hide
      `PartialValueStrategy` and `PartialKeyStrategy` now have namespaced configurations. So use:

      `document.id.strategy.partial.value.projection.type`, `document.id.strategy.partial.value.projection.list`,
      `document.id.strategy.partial.key.projection.type` and `document.id.strategy.partial.key.projection.list`.

      instead of [key|value].projection.[type|list].

      Remove note about {{PROJECTION POST PROCESSORS ARE NOT COMPATIBLE WITH PARTIALVALUESTRATEGY}}
      Show
      `PartialValueStrategy` and `PartialKeyStrategy` now have namespaced configurations. So use: `document.id.strategy.partial.value.projection.type`, `document.id.strategy.partial.value.projection.list`, `document.id.strategy.partial.key.projection.type` and `document.id.strategy.partial.key.projection.list`. instead of [key|value].projection.[type|list]. Remove note about {{PROJECTION POST PROCESSORS ARE NOT COMPATIBLE WITH PARTIALVALUESTRATEGY}}
    • Case:

      Description

      We want to support a solution to config more than one processor on the chain of post processors.

      Consider the following example:

        "topic.override.source.document.id.strategy":"com.mongodb.kafka.connect.sink.processor.id.strategy.PartialValueStrategy",
           "topic.override.source.collection":"sink",
           "topic.override.source.value.projection.type":"whitelist",
           "topic.override.source.value.projection.list":"attuid",
           "topic.override.source.writemodel.strategy": "com.mongodb.kafka.connect.sink.writemodel.strategy.ReplaceOneBusinessKeyStrategy",
       
           "topic.override.source.post.processor.chain":"com.mongodb.kafka.connect.sink.processor.WhitelistValueProjector",
           "topic.override.source.collection":"sink",
           "topic.override.source.value.projection.type":"whitelist",
           "topic.override.source.value.projection.list":"attuid, name, pc",
           "topic.override.source.batch.size":"100","name":"mongo-sink"}
      

      Currently, this fails to update correctly the MongoDB documents. The current design only allows one processor to be configured as it is not possible to configure settings for each of the processors.

      This work is to implement a new id strategy that does not reuse configuration parameters.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              ross.lawley Ross Lawley
              Reporter:
              seth.payne Seth Payne (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: