[KAFKA-185] Support dynamic topic naming strategies Created: 22/Jan/21  Updated: 28/Oct/23  Resolved: 08/Feb/21

Status: Closed
Project: Kafka Connector
Component/s: Source
Affects Version/s: None
Fix Version/s: 1.4.0

Type: Improvement Priority: Major - P3
Reporter: Ross Lawley Assignee: Ross Lawley
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Documented
Duplicate
duplicates KAFKA-184 Configurable topic suffix for source ... Closed
Related
is related to KAFKA-184 Configurable topic suffix for source ... Closed
Epic Link: KAFKA-47
Documentation Changes: Needed
Documentation Changes Summary:

Added Dynamic Topic Mapping for the Source

Added a new interface `TopicMapper` with a `getTopic` method. Implementations can take the Change Stream Document to determine the correct `topic` to publish the data to.

The Source connector includes a single implementation: `DefaultTopicMapper`. The `DefaultTopicMapper` uses the 'ns' field to determine the topic to publish to.

The following configuration options can help configure topic mapping:

  • `topic.prefix`
    Adds a prefix to the topic eg: `<prefix>.<databaseName>.<collectionName>`
    Default: ""
  • `topic.suffix`
    Adds a suffix to the topic eg: `<databaseName>.<collectionName>.<suffix>`
    Default: ""
  • `topic.namespace.map`
    A json map to maps change stream document namespaces to topics:
    For example: ` {\"db\": \"dbTopic\", \"db.coll\": \"dbCollTopic\"}

    ` will map all change stream documents from the `db` database to `dbTopic.<collectionName>` apart from any documents from the `db.coll` namespace which map to the `dbCollTopic` topic.
    Default: ""

  • `topic.mapper` - use an alternative mapping class, users can implement their own `TopicMapper` class to handle the mapping logic.
    Default: com.mongodb.kafka.connect.source.topic.mapping.DefaultTopicMapper

 Description   

Similar to KAFKA-159 but takes the MongoNamespace and maps it to a topic.

Only a default mapper will be added and will include prefix and suffix settings. However, it will allow users to be flexible in



 Comments   
Comment by Githook User [ 08/Feb/21 ]

Author:

{'name': 'Mario Molina', 'email': 'mmolimar@gmail.com', 'username': 'mmolimar'}

Message: Added Dynamic Topic Mapping for the Source

Added a new interface `TopicMapper` with a `getTopic` method. Implementations can take the
Change Stream Document to determine the correct `topic` to publish the data to.

The Source connector includes a single implementation: `DefaultTopicMapper`.
The `DefaultTopicMapper` uses the 'ns' field to determine the topic to publish to. The following
configuration options can help configure topic mapping:

  • `topic.prefix` - Add a prefix to the topic eg: `<prefix>.<databaseName>.<collectionName>`
  • `topic.suffix` - Add a suffix to the topic eg: `<databaseName>.<collectionName>.<suffix>`
  • `topic.namespace.map` - Set a json map to maps change stream document namespaces to topics:
    For example: ` {\"db\": \"dbTopic\", \"db.coll\": \"dbCollTopic\"}

    ` will map all Change Stream
    Documents from the `db` database to `dbTopic.<collectionName>` apart from any documents
    from the `db.coll` namespace which map to the `dbCollTopic` topic.

  • `topic.mapper` - use an alternative mapping class, users can implement their own
    `TopicMapper` class to handle the mapping logic.

KAFKA-185
Branch: master
https://github.com/mongodb/mongo-kafka/commit/5b9a5bd32905bd89a646629c6840411cdae27bec

Generated at Thu Feb 08 09:05:47 UTC 2024 using Jira 9.7.1#970001-sha1:2222b88b221c4928ef0de3161136cc90c8356a66.