[KAFKA-254] Allow Source Connector to Automatically Restart After "Resume Token Not Found" Created: 05/Oct/21  Updated: 28/Oct/23  Resolved: 18/Oct/21

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: None
Fix Version/s: 1.3.0

Type: Improvement Priority: Unknown
Reporter: Diego Rodriguez (Inactive) Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: internal-user
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
Case:
Documentation Changes: Needed
Documentation Changes Summary:

If implemented, this new behavior will need to be properly documented.


 Description   

Hi Team,

Right now, when the Source Connector encounters a "Resume Token Not Found" error it fails and doesn't automatically recover. The reasoning behind this decision is that there are data loss implications when this happens and the safe approach is to stop, allow the user to understand the implications, and carefully decide the next steps.

Would it be possible to make this more flexible by adding a configuration option that allows the Source Connector to automatically restart from scratch whenever a "Resume Token Not Found" error is found?

A clear message that this happened and the implications should be written to the logs, as well as a startup banner that states the currently configured behavior and its implications.

Thanks
Diego



 Comments   
Comment by Robert Walters [ 18/Oct/21 ]

Fixed as part of error tolerance work in 1.3

Comment by Ross Lawley [ 11/Oct/21 ]

Hi diego.rodriguez,

With added support for errors.tolerance=all in KAFKA-105, support for missing / invalid resume tokens and auto restarting without a resumeToken was added. KAFKA-215 then added support for connector specific error settings mongo.errors.tolerance=all so that only connector based errors would be tolerated.

Does that setting meet the need for the user?

Ross

Generated at Thu Feb 08 09:05:57 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.