[KAFKA-208] Error on heartbeat setup with existing SMT Created: 16/Mar/21 Updated: 27/Oct/23 Resolved: 29/Mar/21 |
|
| Status: | Closed |
| Project: | Kafka Connector |
| Component/s: | Source |
| Affects Version/s: | 1.3.0 |
| Fix Version/s: | None |
| Type: | Bug | Priority: | Major - P3 |
| Reporter: | Vlad Goldman | Assignee: | Ross Lawley |
| Resolution: | Works as Designed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Issue Links: |
|
||||
| Description |
|
When trying to add a heartbeatĀ to the existing working configuration
I'm getting the following error:
|
| Comments |
| Comment by Tasos Zervos [ 12/Sep/23 ] | ||||||||||
|
vlad@guesty.com - have you found a resolution to this last issue? | ||||||||||
| Comment by Vlad Goldman [ 30/Dec/21 ] | ||||||||||
|
@ross.lawley thank you for the great suggestion with predicate, we are using it in production since that time. Although now we faced another problem that seems to be relevant, in case if we want to add heartbeat to a connect that using schema registry and subject name strategy like:
we do get the following error:
Is there any workaround can be applied or we need to implement custom serializer that will skip heartbeat messages? | ||||||||||
| Comment by Ross Lawley [ 29/Mar/21 ] | ||||||||||
|
Hi vlad@guesty.com, As described above SMT's apply to all messages created by a connector. The issue is the Heartbeat messages don't match the requirements for the SMTs. However, there is good news you can use a predicate to only apply the SMT's to the messages you want to.
In the above example we match all topics starting with "prefix", you could easily switch that to use negate to discount all messages to the "heartbeats-mongodb" topic. I'm closing this ticket as "works as designed" as SMT's are applied after the connector passes the messages to Kafka connect. All the best, Ross | ||||||||||
| Comment by Githook User [ 29/Mar/21 ] | ||||||||||
|
Author: {'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}Message: Ensure the heartbeat message schema is optional A non optional schema breaks value conversion.
| ||||||||||
| Comment by Ross Lawley [ 24/Mar/21 ] | ||||||||||
|
Hi vlad@guesty.com, The SMT mechanism is a layer on top of the Source Connector API. So I imagine the SMT blanket applies to all SourceRecords produced by a Source Connector. Ross | ||||||||||
| Comment by Vlad Goldman [ 24/Mar/21 ] | ||||||||||
|
@ross.lawley but why SMT even applied to heartbeat records? Shouldn't it just skip them? | ||||||||||
| Comment by Ross Lawley [ 17/Mar/21 ] | ||||||||||
|
Hi vlad@guesty.com, Thanks for the ticket. The heartbeat mechanism provides differently shaped SourceRecords to the configured heartbeat topic. As such SMT's would need to be able to handle this scenario.
Ross |