[KAFKA-292] record will lose if offset submitted earlier Created: 16/Feb/22  Updated: 27/Oct/23  Resolved: 16/Feb/22

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Critical - P2
Reporter: Liu Zhiqiang Assignee: Ross Lawley
Resolution: Gone away Votes: 0
Labels: external-user
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

In the class-[org|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org].[apache|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache].[kafka|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka].[connect|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka.connect].[runtime|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka.connect.runtime].[WorkerSourceTask|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka.connect.runtime%7BWorkerSourceTask.java%E2%98%83WorkerSourceTask]
enter function sendRecords() 
we know that: 
 
firstly, execute 
-
// Offsets are converted & serialized in the OffsetWriter
offsetWriter.offset(record.sourcePartition(), record.sourceOffset());
-
then execute
producer.send(...)
 
but if producer.send(...) fail, but the offset of the record has been submit to offsetWriter
 
and we know that, 
[org|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org].[apache|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache].[kafka|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka].[connect|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka.connect].[runtime|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka.connect.runtime].[SourceTaskOffsetCommitter|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka.connect.runtime%7BSourceTaskOffsetCommitter.java%E2%98%83SourceTaskOffsetCommitter].schedule([ConnectorTaskId|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka.connect.runtime%7BSourceTaskOffsetCommitter.java%E2%98%83SourceTaskOffsetCommitter~schedule~QConnectorTaskId;~QWorkerSourceTask;%E2%98%82ConnectorTaskId] id, [WorkerSourceTask|eclipse-javadoc:%E2%98%82=kafka-src-3.0.0/src%3Cconnect.runtime.src.main.java.org.apache.kafka.connect.runtime%7BSourceTaskOffsetCommitter.java%E2%98%83SourceTaskOffsetCommitter~schedule~QConnectorTaskId;~QWorkerSourceTask;%E2%98%82WorkerSourceTask] workerTask) is invoked periodically,
then the fail record has been recrod to the default file :  /tmp/connect.offsets
 
so it is wrong, Is this a bug?
 



 Comments   
Comment by Ross Lawley [ 16/Feb/22 ]

Closed as requested.

Comment by Liu Zhiqiang [ 16/Feb/22 ]

please close this one, content is wrong, I will create another one! thank you

Generated at Thu Feb 08 09:06:02 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.