[KAFKA-138] Improve resume change stream error message Created: 06/Aug/20  Updated: 27/Oct/23  Resolved: 17/Aug/20

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Improvement Priority: Major - P3
Reporter: Andrey B Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

In https://jira.mongodb.org/browse/KAFKA-91 added error message.
I think it's not completely correct.
https://github.com/mongodb/mongo-kafka/blob/master/src/main/java/com/mongodb/kafka/connect/source/MongoSourceTask.java#L300-L301

Kafka Connect uses its own mechanism, not kafka consumer group to store offsets for source connectors. So, as far as I understand there is no way to use kafka-consumer-groups.sh to reset offsets.

If I'm wrong about using kafka-consumer-groups.sh, please make the message more explicit.

Could be helpful:
https://rmoff.net/2019/08/15/reset-kafka-connect-source-connector-offsets/



 Comments   
Comment by Andrey B [ 17/Aug/20 ]

Hi, my mistake. I wasn't clear enough.
I spoke only about source connectors.

I mean Source connector usesĀ OffsetStorageReader and OffsetStorageWriter to work with offsets. So, there is no consumer group and kafka-consumer-groups.sh can't be used.

See more: https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageWriter.java#L39-L43

Comment by Ross Lawley [ 17/Aug/20 ]

Hi andreworty@gmail.com,

Offsets are handled by Kafka outside of the connector. The connector itself just declares an offset when producing SourceRecords. This allows for changes in Kafka topologies and connector restarts. As far as I'm aware the Kafka connect uses the existing Kafka offset mechanisms and as such you can either delete the offset file when running standalone or manage the meta data in the offset topic via the mentioned shell script or manually.

As KAFKA-134 will also look into making errors, missing offsets etc easier to deal with, I'm closing this ticket.

Generated at Thu Feb 08 09:05:40 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.