[KAFKA-176] Improve heartbeat usability Created: 26/Nov/20 Updated: 27/Oct/23 Resolved: 15/Mar/21 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | Source |
| Affects Version/s: | None |
| Fix Version/s: | 1.5.0 |
| Type: | Improvement | Priority: | Major - P3 |
| Reporter: | Ross Lawley | Assignee: | Ross Lawley |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||||||
| Case: | (copied to CRM) | ||||||||
| Description |
|
Currently, users have to consume heartbeat messages themselves manually. Using KSql, custom code or the console script:
This is not very user friendly and we should provide a way to do this easily. Perhaps a background thread in the Source could handle it or a custom Sink handler. |
| Comments |
| Comment by Githook User [ 31/Mar/21 ] |
|
Author: {'name': 'Nathan Leniz', 'email': 'nathan.leniz@mongodb.com', 'username': 'terakilobyte'}Message: remove wording that users must consume heartbeat per |
| Comment by Githook User [ 15/Mar/21 ] |
|
Author: {'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}Message: Updated Heartbeat test Shows that heartbeats don't have to be consumed.
|
| Comment by Ross Lawley [ 15/Mar/21 ] |
|
After reviewing the implementation and how Kafka works internally, it has been shown that this assumption is incorrect. Heartbeats don't have to be read from the topic to be set as the last produced offset from the OffsetStorageReader. tldr; even without reading heartbeat offsets, restarting the connector will use the last produced resumeToken (either from a changestream document or from the post batch resume token from the changestream cursor). |