[KAFKA-14] Simplify config topic/collection mapping and validation Created: 14/Mar/19  Updated: 28/Oct/23  Resolved: 29/Mar/19

Status: Closed
Project: Kafka Connector
Component/s: None
Affects Version/s: None
Fix Version/s: 0.1

Type: Task Priority: Major - P3
Reporter: Ross Lawley Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

Currently mapping is confusing:

#Kafka topics to consume from
topics=foo-t,blah-t
 
#MongoDB collections to write to
mongodb.collections=foo-c,blah-c
 
#Named topic <-> collection mappings
mongodb.collection.foo-t=foo-c
mongodb.collection.blah-t=blah-c
 
 
# specific processing settings for topic 'foo-t' -> collection 'foo-c'
mongodb.document.id.strategy.foo-c=c.m.k.c.processor.id.strategy.UuidStrategy
mongodb.post.processor.chain.foo-c=c.m.k.c.processor.DocumentIdAdder,c.m.k.c.processor.BlacklistValueProjector
mongodb.value.projection.type.foo-c=blacklist
mongodb.value.projection.list.foo-c=k2,k4 
mongodb.max.batch.size.foo-c=100

Theres a disconnect:

#MongoDB collections to write to
mongodb.collections=foo-c,blah-c
 
# topic <-> collection mapping
mongodb.collection.<*topicName*>=<collectionName>
 
# topic <-> collection overrides
mongodb.<configuration>.<*collectionName*>=someValue

I propose simplifying the configuration, so to do without mongodb.collections. I also propose making the configuration topic centric rather than collection - as topics are required.

topics=foo-t,blah-t
 
# specific processing settings for topics
mongodb.overrides.<topicName>.<config>=someValue

Everything apart from the topics list and the connection String can be overridden at a topic level.


Generated at Thu Feb 08 09:05:21 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.