Uploaded image for project: 'Kafka Connector'
  1. Kafka Connector
  2. KAFKA-368

MongoDB source connector (>1.9.0) does not release connections

    • Type: Icon: Bug Bug
    • Resolution: Duplicate
    • Priority: Icon: Unknown Unknown
    • None
    • Affects Version/s: 1.9.0
    • Component/s: None
    • None
    • Not Needed

      Our company faced an issue after migrating to the MongoDB source connector version 1.9.0 (and higher). Description:
      We have an Atlas Cloud MongoDB non-production cluster that is pausing by schedule (at night and at weekends) due to operational costs. Also, we have a self-hosted kafka-connect cluster with an installed MongoDB source connector in there. There's also a self-hosted application that monitors all the connectors by schedule (i.e. every 30 seconds) and restarts all the failed connectors and tasks. Before the upgrade to 1.9.0, we used version 1.8.1 and everything worked as expected - during the MongoDB downtime (at night) source connector keeps failing and restarted by the app. When the cluster is up again, the number of connections remains roughly the same (see screenshot #1 attached).

      After we migrated to version 1.9.0 (the issue appears also in all the versions up to 1.10.0), the number of connections started to grow each time MongoDB is down at night (see screenshot #2 attached). The connections are kept alive during the time and eventually, it leads to the reaching of the Atlas cluster limits. After that, our connector is not able to consume any changes from MongoDB and write to Kafka. 
      As a quick fix, we had to stop all the connectors and redeploy kafka-connect cluster.

      We've tried various MongoDB Java driver configs such as connectTimeoutM, ocketTimeoutMS, maxIdleTimeMS, maxLifeTimeMS but it had no effect on the number of connections. 

      Versions that we use:

      • Atlas MongoDB cluster - 5.0.17, 6.0
      • Apache Kafka - 3.4.0
      • kafka-connect cluster is built based on JDK 17

      The only difference between the 2 use cases was MongoDB Source connector (1.8.1 vs 1.9.0).

        1. 1. mongodb-source-connector-1.8.1-downtime.png
          24 kB
          Artsiom Krauchanka
        2. 2. mongodb-source-connector-1.9.0-downtime.png
          23 kB
          Artsiom Krauchanka

            Assignee:
            valentin.kovalenko@mongodb.com Valentin Kavalenka
            Reporter:
            artsiom.krauchanka@comparethemarket.com Artsiom Krauchanka
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: