-
Type: Improvement
-
Resolution: Unresolved
-
Priority: Unknown
-
None
-
Affects Version/s: 1.6.1
-
Component/s: Source
-
None
When setting the copy.existing option to true, the source connector first performs a copy of the whole MongoDB collection. However, if the connector restarts before the copy is finished, then the copy restarts from scratch.
This is an issue for large collections that take a lot of time to ingest because:
- The probability of a restart happening during the copy is more important
- The impact is more important: a lot of duplicates are written to Kafka and the process takes more time to finish than expected.
It would be great if there was some failure recovery mechanism, that could make sure that the copy resumes from where it left off before the restart of the connector.
- depends on
-
KAFKA-61 Improve Source Connector Performance / Scalability
- Backlog