[KAFKA-176] Improve heartbeat usability Created: 26/Nov/20  Updated: 27/Oct/23  Resolved: 15/Mar/21

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: None
Fix Version/s: 1.5.0

Type: Improvement Priority: Major - P3
Reporter: Ross Lawley Assignee: Ross Lawley
Resolution: Works as Designed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Related
related to KAFKA-76 Reuse the postBatchResumeToken Closed
Case:

 Description   

Currently, users have to consume heartbeat messages themselves manually.

Using KSql, custom code or the console script:

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic __mongodb_heartbeats --skip-message-on-error

This is not very user friendly and we should provide a way to do this easily.

Perhaps a background thread in the Source could handle it or a custom Sink handler.



 Comments   
Comment by Githook User [ 31/Mar/21 ]

Author:

{'name': 'Nathan Leniz', 'email': 'nathan.leniz@mongodb.com', 'username': 'terakilobyte'}

Message: remove wording that users must consume heartbeat

per KAFKA-176, consuming heartbeats is not required
Branch: master
https://github.com/10gen/docs-kafka-connector/commit/e2b457060a925ece85320209a5d76f99222e6482

Comment by Githook User [ 15/Mar/21 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Updated Heartbeat test

Shows that heartbeats don't have to be consumed.

KAFKA-176
Branch: master
https://github.com/mongodb/mongo-kafka/commit/e5ff9b899f7bcde76d1f2d0fd3d449a9819ce242

Comment by Ross Lawley [ 15/Mar/21 ]

After reviewing the implementation and how Kafka works internally, it has been shown that this assumption is incorrect.

Heartbeats don't have to be read from the topic to be set as the last produced offset from the OffsetStorageReader.

tldr; even without reading heartbeat offsets, restarting the connector will use the last produced resumeToken (either from a changestream document or from the post batch resume token from the changestream cursor).

Generated at Thu Feb 08 09:05:45 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.