[KAFKA-293] record will be lost if offset submitted earlier Created: 16/Feb/22  Updated: 27/Oct/23  Resolved: 15/Mar/22

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Question Priority: Major - P3
Reporter: Liu Zhiqiang Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: external-user
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

In the class - org.apache.kafka.connect.runtime.WorkerSourceTask
enter function sendRecords()
we know that:

firstly, execute
-
// Offsets are converted & serialized in the OffsetWriter
offsetWriter.offset(record.sourcePartition(), record.sourceOffset());
-
then execute
producer.send(...)

but if producer.send(...) fail, but the offset of the record has been submit to offsetWriter

and we know that,
org.apache.kafka.connect.runtime.SourceTaskOffsetCommitter.schedule(ConnectorTaskId id, WorkerSourceTask workerTask)  is invoked periodically!!!
then it is possible that the offset of fail record has been recroded into the default file : /tmp/connect.offsets

so it is wrong

Is this a bug?



 Comments   
Comment by Ross Lawley [ 15/Mar/22 ]

Hi 837500869@qq.com,

Thanks for the ticket, the offset mechanism is part of Kafka's resilience and how the Kafka connect framework works with it is outside the remit of the MongoDB Kafka Connector.

I think your question is better suited to the Kafka mailing list / development team.

All the best,

Ross

Comment by Liu Zhiqiang [ 25/Feb/22 ]

After I read the code of  kafka connect - mongodb source ,

I have rewrited it by myself ,

please see https://github.com/lzqdename/pulsar-mongodb-source-connector

can everyone give me some improve suggestions? 

Thank you very much !

 

Comment by Esha Bhargava [ 22/Feb/22 ]

837500869@qq.com Thank you for reporting this issue! We'll look into it and get back to you soon.

Comment by Liu Zhiqiang [ 16/Feb/22 ]

title is wrong, should be "record will be lost if offset submitted earlier"

Comment by Liu Zhiqiang [ 16/Feb/22 ]

Affected version-Mongo Kafka Connector 1.6.1

Generated at Thu Feb 08 09:06:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.