[KAFKA-158] Add offset.partition.name configuration Created: 18/Sep/20  Updated: 28/Oct/23  Resolved: 30/Sep/20

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: None
Fix Version/s: 1.3.0

Type: Improvement Priority: Major - P3
Reporter: Ross Lawley Assignee: Ross Lawley
Resolution: Fixed Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
Case:
Documentation Changes: Needed

 Description   

Add new configuration offset.partition.name which will be used instead of the default naming method if defined.
In such a case when the operations team wants a fresh start they just have to restart the connector with a new offset.partition.name configuration.

-Was: Add ignore previous offsets configuration

Add a new configuration to ignore any saved offsets.

This would be useful during reconfiguration of the Source connector, so that a fresh start can be mandated by the operations team.-



 Comments   
Comment by Githook User [ 29/Sep/20 ]

Author:

{'name': 'Ross Lawley', 'email': 'ross.lawley@gmail.com', 'username': 'rozza'}

Message: Added `offset.partition.name` configuration. (#41)

Allows for custom partition naming schemes.

KAFKA-158
Branch: master
https://github.com/mongodb/mongo-kafka/commit/c3c35bc61fc996ef1a54616651cb9c8ab4db548e

Comment by Ross Lawley [ 22/Sep/20 ]

https://github.com/mongodb/mongo-kafka/pull/41

Comment by Ross Lawley [ 21/Sep/20 ]

Hi andreworty@gmail.com,

Great idea, seems much simpler to manage for the ops team and far more robust!

Many thanks!

Ross

Comment by Andrey B [ 21/Sep/20 ]

Hi Ross

I just wanted to share some thoughts about this ticket.

I think about configuration offset.partition.name which will be used instead of createNamespaceString method's result if defined.
In such a case when the operations team wants a fresh start they just have to restart the connector with a new offset.partition.name configuration.

Adding configuration to ignore previous offsets (e.g. ignore.previous.offset configuration with true/false values) could lead to an unexpected fresh start on the connector's restart or rebalancing.

 

Thanks

Generated at Thu Feb 08 09:05:43 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.