Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-315

Resume copy from where it left off after a restart

    • Type: Icon: Improvement Improvement
    • Resolution: Unresolved
    • Priority: Icon: Unknown Unknown
    • None
    • Affects Version/s: 1.6.1
    • Component/s: Source
    • Labels:
      None

      When setting the copy.existing option to true, the source connector first performs a copy of the whole MongoDB collection. However, if the connector restarts before the copy is finished, then the copy restarts from scratch.

      This is an issue for large collections that take a lot of time to ingest because:

      • The probability of a restart happening during the copy is more important
      • The impact is more important: a lot of duplicates are written to Kafka and the process takes more time to finish than expected.

      It would be great if there was some failure recovery mechanism, that could make sure that the copy resumes from where it left off before the restart of the connector.

            Assignee:
            Unassigned Unassigned
            Reporter:
            colin.smetz@euranova.eu Colin Smetz
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: