Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-339

Topic names cleanup to match Kafka topic naming restrictions

    XMLWordPrintableJSON

Details

    • Icon: New Feature New Feature
    • Resolution: Unresolved
    • Icon: Major - P3 Major - P3
    • 1.12.0
    • None
    • None
    • None

    Description

      By default, the MongoDB Kafka source connector publishes change event data to a Kafka topic with the same name as the MongoDB namespace from which the change events originated. 

      As naming restrictions for database/collection names in MongoDB allow characters that are restricted for Kafka topics (e.g whitespace), such namespaces will fail in the connector.

      While topic.namespace.map can be used as a workaround, it won't be of any help for newly created databases/collections and has to be constantly updated upon failures.

      The property topic.mapper looks relevant but lacks any documentation and seems complicated to implement.

      IMO, a topic name sanitization should be implemented, something similar to how it's done in Debezium:  https://github.com/debezium/debezium/blob/4d577655968d2e00edb67a7e53702b8c38a17023/debezium-api/src/main/java/io/debezium/spi/topic/TopicNamingStrategy.java#L46

      (Debezium MongoDB Source Connector cleans up any illegal characters for topic name by converting them to underscores)

      Attachments

        Activity

          People

            Unassigned Unassigned
            alonp@accessfintech.com Alon Prantsipal
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: