Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-117

Is it possible to implement copying new collections if pipeline has been changed?

    XMLWordPrintableJSON

Details

    • Icon: New Feature New Feature
    • Resolution: Won't Fix
    • Icon: Major - P3 Major - P3
    • None
    • None
    • Source
    • None

    Description

      Use case workflow:

      Create connector with config:

      "pipeline": "[{\"$match\": {\"ns.coll\": {\"$regex\": /^(col1|col2)$/}}}]",
      "copy.existing": "true",
      ... 
      

      After some time update pipeline in connector's config to:

      "[{\"$match\": {\"ns.coll\": {\"$regex\": /^(col1|col2|col3)$/}}}]"

      Desired result after restart:

      • save resume token
      • somehow understand that need to copy only documents from 'col3' collection
      • copy documents from 'col3'
      • start streaming from saved resume token

       

      What do you think about it?

      Attachments

        Activity

          People

            ross@mongodb.com Ross Lawley
            andreworty@gmail.com Andrey B
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: