Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-339

Topic names cleanup to match Kafka topic naming restrictions

    • Type: Icon: New Feature New Feature
    • Resolution: Unresolved
    • Priority: Icon: Major - P3 Major - P3
    • 2.0.0
    • Affects Version/s: None
    • Component/s: None
    • Labels:
      None

      By default, the MongoDB Kafka source connector publishes change event data to a Kafka topic with the same name as the MongoDB namespace from which the change events originated. 

      As naming restrictions for database/collection names in MongoDB allow characters that are restricted for Kafka topics (e.g whitespace), such namespaces will fail in the connector.

      While topic.namespace.map can be used as a workaround, it won't be of any help for newly created databases/collections and has to be constantly updated upon failures.

      The property topic.mapper looks relevant but lacks any documentation and seems complicated to implement.

      IMO, a topic name sanitization should be implemented, something similar to how it's done in Debezium:  https://github.com/debezium/debezium/blob/4d577655968d2e00edb67a7e53702b8c38a17023/debezium-api/src/main/java/io/debezium/spi/topic/TopicNamingStrategy.java#L46

      (Debezium MongoDB Source Connector cleans up any illegal characters for topic name by converting them to underscores)

            Assignee:
            Unassigned Unassigned
            Reporter:
            alonp@accessfintech.com Alon Prantsipal
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: