Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-121

MongoDB Kafka source connector always has single task

XMLWordPrintableJSON

    • Type: Icon: Bug Bug
    • Resolution: Works as Designed
    • Priority: Icon: Major - P3 Major - P3
    • None
    • Affects Version/s: 1.1
    • Component/s: Source
    • Labels:
      None
    • Environment:
      Windows 10, 8GB RAM, AMD Ryzen 5 2500U, 64Bit, HDD
    • Needed
    • Hide

      If I set tasks.max=10, using single topic(three partitions), I still see only one task while the fail over to another worker is working fine.
      Spawning(when and how) of source connector "tasks" is not mentioned in documentation of Official MongoDB source connector. Is this the expected behavior? If yes, then documentation should mention that.

      Show
      If I set tasks.max=10, using single topic(three partitions), I still see only one task while the fail over to another worker is working fine. Spawning(when and how) of source connector "tasks" is not mentioned in documentation of Official MongoDB source connector. Is this the expected behavior? If yes, then documentation should mention that.

      I am trying to setup MongoDB-Kafka-Connect as Source listening to change-stream of one collection on my windows machine. I am running three worker nodes on localhost windows machine on ports 8083, 8084, 8085.

      I am able to create connector with one task and receive change stream events successfully and fail lover to other worker node is also working fine. But the number of tasks spawned is always one. I have set tasks.max=10, and used multiple threads to generate high volume of change stream events but even then number of task remains one.

      This kind of makes Kafka-producer(MongoDB source connector) not scalable at all. While with multiple partitions on the same topic, consumers are scalable. Is this an issue that I am experiencing or MongoDB-Kafka-Source connector is designed this way?

            Assignee:
            ross@mongodb.com Ross Lawley
            Reporter:
            jawaid.hamid@gmail.com Hamid Jawaid
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: