Is it possible to implement copying new collections if pipeline has been changed?

XMLWordPrintableJSON

    • Type: New Feature
    • Resolution: Won't Fix
    • Priority: Major - P3
    • None
    • Affects Version/s: None
    • Component/s: Source
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Use case workflow:

      Create connector with config:

      "pipeline": "[{\"$match\": {\"ns.coll\": {\"$regex\": /^(col1|col2)$/}}}]",
      "copy.existing": "true",
      ... 
      

      After some time update pipeline in connector's config to:

      "[{\"$match\": {\"ns.coll\": {\"$regex\": /^(col1|col2|col3)$/}}}]"

      Desired result after restart:

      • save resume token
      • somehow understand that need to copy only documents from 'col3' collection
      • copy documents from 'col3'
      • start streaming from saved resume token

       

      What do you think about it?

            Assignee:
            Ross Lawley
            Reporter:
            Andrey B
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved: