ExportXMLWordPrintableJSON

    • Type: Improvement
    • Resolution: Fixed
    • Priority: Unknown
    • 3.1.3
    • Affects Version/s: 3.1.2
    • Component/s: Source
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      We ran into this while testing the source connector for CDC. If you change the set of hosts in connection.uri and restart the connector, it no longer finds its stored offset. It then starts from startup.mode (latest by default) and every change made while it was down is skipped. The task stays RUNNING and nothing is logged above INFO, so the gap only shows up later downstream.

      The cause is that when offset.partition.name is not set, the offset partition is built from the hosts in the connection string (MongoSourceTask.createDefaultPartitionName). Different hosts give a different partition key, the old offset is never read, and getResumeToken() returns null.

      We know offset.partition.name avoids this (KAFKA-158). The problem is that the default is tied to something people change for unrelated reasons, like replacing a replica set member or moving from hostnames to IPs or new DNS names, and nothing tells you the offset is gone.

      Versions: mongo-kafka 3.1.2 (same code on master), Kafka Connect 3.7.0 distributed, MongoDB 7.0 replica set, JDK 11.

      Steps to reproduce

      1. Single node replica set, connector watching a database: connection.uri=mongodb://mongo:27017/?replicaSet=rs0, database=vl, startup.mode=latest, offset.partition.name not set.
      2. Run a steady update load (we used about 1,200 updates/s) and let the connector catch up.
      3. Delete the connector and create it again with the same name, pointing at the same server by IP: connection.uri=mongodb://172.24.0.2:27017/?replicaSet=rs0.
      4. Count change events in the topic against updates written. Each update carries an incrementing ver field, so missing events can be identified exactly.

      What we saw (each run about 147k updates, connector down for roughly 5 seconds)

      Change to connection.uri offset.partition.name Missing events
      hostname replaced with IP (3 runs) not set 6,208 / 6,446 / 6,698
      extra host added (mongo:27017,172.24.0.2:27017) not set 6,224
      hostname replaced with IP via PUT /connectors/<name>/config not set 119 (shorter restart, same behaviour)
      hostname replaced with IP set 0
      same via PUT set 0
      hosts only reordered not set 0 (the driver sorts hosts, so the key does not change)
      only a URI option changed (appName) not set 0

      After the change, GET /connectors/<name>/offsets shows two partitions for the same connector, for example mongodb://mongo:27017/vl and mongodb://172.24.0.2:27017,mongo:27017/vl. The old offset is still there, it just isn't read any more. No WARN or ERROR is logged at the point where the stored offset is missing and the connector falls back to startup.mode.

      For comparison, the Debezium MongoDB connector keeps its offset across the same change, because it keys offsets on topic.prefix.

      Not tested, from reading the code: with a mongodb+srv:// URI only the SRV hostname goes into the key, so member changes behind SRV should be fine but changing the SRV name would hit the same issue. Replacing replica set members on the server side, without editing connection.uri, should not affect the key.

      Expected
      At minimum, the connector should not silently skip data because the connection string was edited. Any of these would help:

      1. Log a WARN when no stored offset is found and the connector falls back to startup.mode, including the partition name it looked up. Small change in getResumeToken(); we're happy to send a PR.
      2. An opt-in setting to fail the task instead of falling back when no offset is found, so a changed key can't skip data quietly.
      3. A note in the docs for offset.partition.name that the default depends on the hosts in connection.uri, recommending it be set for production.

      Longer term, a default that doesn't depend on the host list (the replica set name, for example) would avoid the problem. That changes existing keys and would need a fallback to the old key, so we'd leave the approach to you.

      Workaround: always set offset.partition.name.

      We can share the docker-compose setup and scripts we used.

            Assignee:
            Andrea Rahmanan
            Reporter:
            Dhairya Gupta (EXT)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: